Oct 30 18:36:04.424874 ip-10-0-128-92 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:36:04.886962 ip-10-0-128-92 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:04.886962 ip-10-0-128-92 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:36:04.886962 ip-10-0-128-92 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:04.886962 ip-10-0-128-92 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:04.891267 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891189    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:04.891267 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891268    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891272    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891275    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891278    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891281    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891284    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891287    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891289    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891292    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891294    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891297    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891301    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891303    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891306    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891309    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891311    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891314    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891317    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891320    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:04.891333 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891322    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891331    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891336    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891340    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891343    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891346    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891349    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891352    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891354    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891357    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891359    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891362    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891364    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891367    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891370    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891372    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891375    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891377    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891380    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891383    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:04.891858 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891385    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891387    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891390    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891392    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891395    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891397    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891401    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891404    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891406    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891409    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891412    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891415    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891418    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891421    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891424    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891426    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891429    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891431    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891434    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891436    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:04.892319 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891439    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891442    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891444    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891446    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891449    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891452    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891454    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891457    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891459    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891462    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891464    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891467    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891470    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891472    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891475    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891477    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891481    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891484    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891487    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891490    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:04.892814 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891493    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891495    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891498    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891500    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891504    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891508    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891511    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891514    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891516    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891519    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891522    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891524    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891527    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891530    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891533    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891535    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891538    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891541    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:04.893284 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891543    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891546    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891549    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891551    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891555    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891557    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891560    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.891563    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892355    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892362    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892365    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892368    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892371    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892374    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892377    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892380    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892382    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892385    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892388    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892390    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:04.893727 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892393    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892396    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892398    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892401    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892404    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892406    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892409    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892411    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892414    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892417    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892419    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892422    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892424    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892427    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892430    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892432    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892435    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892438    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892440    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892443    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:04.894228 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892446    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892449    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892451    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892454    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892457    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892460    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892463    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892465    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892468    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892471    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892473    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892476    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892479    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892481    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892484    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892488    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892491    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892494    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892497    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:04.894775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892499    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892502    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892504    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892507    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892509    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892512    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892514    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892517    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892519    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892522    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892525    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892527    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892530    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892533    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892536    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892538    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892542    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892545    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892548    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:04.895231 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892551    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892553    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892556    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892559    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892562    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892565    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892568    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892571    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892573    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892576    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892595    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892598    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892600    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892603    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892608    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892612    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892616    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892619    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892621    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:04.895695 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892624    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892627    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892629    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892632    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892635    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892638    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892640    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892643    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892646    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892650    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892653    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892656    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892658    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892661    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892663    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892666    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.892668    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893951    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893959    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893963    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:04.896171 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893966    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893969    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893972    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893974    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893977    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893981    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893983    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893986    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893989    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893992    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893994    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893997    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.893999    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894002    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894005    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894007    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894010    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894013    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894016    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:04.896666 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894019    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894021    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894025    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894028    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894030    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894033    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894035    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894038    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894041    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894043    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894046    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894048    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894051    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894053    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894056    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894059    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894062    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894064    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894067    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894070    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:04.897118 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894073    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894075    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894078    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894081    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894083    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894086    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894089    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894091    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894093    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894098    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894102    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894105    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894109    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894112    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894115    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894119    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894123    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894125    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894128    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:04.897589 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894131    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894133    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894136    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894139    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894141    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894144    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894146    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894149    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894152    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894154    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894157    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894159    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894162    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894166    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894169    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894171    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894174    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894177    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894179    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894182    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:04.898042 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894185    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894187    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894192    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894195    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894197    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894214    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894218    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894221    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894224    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894227    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894230    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894233    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894236    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894239    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894242    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894245    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894247    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894250    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894252    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:04.898506 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894255    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894258    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894260    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894263    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894265    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.894268    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895289    2571 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895299    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895307    2571 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895313    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895318    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895321    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895325    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895329    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895333    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895337    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895340    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895344    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895347    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895350    2571 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895352    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895356    2571 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895359    2571 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:36:04.898961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895362    2571 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895367    2571 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895370    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895373    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895376    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895380    2571 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895384    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895387    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895391    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895394    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895397    2571 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895400    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895403    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895406    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895409    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895414    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895417    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895420    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895423    2571 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895426    2571 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895429    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895438    2571 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895441    2571 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895444    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:36:04.899495 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895447    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895450    2571 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895454    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895457    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895460    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895464    2571 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895467    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895470    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895473    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895478    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895481    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895484    2571 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895486    2571 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895491    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895494    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895497    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895500    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895504    2571 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895507    2571 flags.go:64] FLAG: --help="false"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895510    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895513    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895516    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895519    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:36:04.900076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895523    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895526    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895530    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895533    2571 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895536    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895539    2571 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895542    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895545    2571 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895548    2571 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895551    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895554    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895557    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895560    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895562    2571 flags.go:64] FLAG: --lock-file=""
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895565    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895568    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895571    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895587    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895591    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895594    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895597    2571 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895600    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895604    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895607    2571 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895610    2571 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:36:04.900684 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895615    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895618    2571 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895622    2571 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895625    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895629    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895632    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895635    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895638    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895641    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895644    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895651    2571 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895654    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895657    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895660    2571 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895663    2571 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895666    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895669    2571 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895672    2571 flags.go:64] FLAG: --port="10250"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895675    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895678    2571 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-03680e6095307cc3e"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895681    2571 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895684    2571 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895687    2571 flags.go:64] FLAG: --register-node="true"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895690    2571 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895694    2571 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895697    2571 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:36:04.901282 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895700    2571 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895703    2571 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895708    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895711    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895714    2571 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895717    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895720    2571 flags.go:64] FLAG: --runonce="false"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895723    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895726    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895729    2571 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895732    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895735    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895738    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895743    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895747    2571 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895750    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895753    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895756    2571 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895759    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895762    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895765    2571 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895768    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895773    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895776    2571 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895780    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:36:04.901914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895784    2571 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895787    2571 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895790    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895793    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895796    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895799    2571 flags.go:64] FLAG: --v="2"
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895804    2571 flags.go:64] FLAG: --version="false"
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895808    2571 flags.go:64] FLAG: --vmodule=""
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895812    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.895816    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895954    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895958    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895963    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895965    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895968    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895971    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895974    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895976    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895979    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895982    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895986    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:04.902510 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895988    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895991    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895993    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895996    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.895999    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896002    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896004    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896007    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896010    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896012    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896015    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896018    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896020    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896023    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896025    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896028    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896031    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896034    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896036    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896039    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:04.903368 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896042    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896045    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896048    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896051    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896054    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896057    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896060    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896062    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896065    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896067    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896070    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896074    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896076    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896079    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896081    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896084    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896088    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896091    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896094    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896096    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:04.904285 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896099    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896102    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896105    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896108    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896110    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896113    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896116    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896118    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896121    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896124    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896127    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896129    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896132    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896134    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896137    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896141    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896144    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896146    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896149    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:04.905126 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896152    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896156    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896159    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896162    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896166    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896169    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896172    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896175    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896178    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896181    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896184    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896186    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896189    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896192    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896194    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896197    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896199    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896202    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896205    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:04.905987 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896207    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896210    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896212    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896215    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896219    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896223    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896226    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896229    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896232    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896237    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896240    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896243    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896246    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896249    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896252    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896254    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:04.906882 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896257    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.896285    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896321    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896325    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896328    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896331    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896333    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896336    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896339    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896342    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896346    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896349    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896352    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896355    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896358    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:04.907378 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896361    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896363    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896366    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896368    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896371    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896374    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896376    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896379    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896381    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896384    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896388    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896392    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896394    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896397    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896400    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896402    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896405    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896408    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896410    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:04.907850 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896414    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896417    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896420    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896422    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896425    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896427    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896430    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896432    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896435    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896438    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896440    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896443    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896445    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896448    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896451    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896453    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896456    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896459    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896461    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896464    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:04.908704 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896467    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896469    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896472    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896476    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896478    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896481    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896484    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896487    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896489    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896492    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896494    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896497    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896500    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896503    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896506    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896508    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896511    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896514    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896517    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896519    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:04.909371 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896523    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896526    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896528    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896531    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896534    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896536    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896539    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896542    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896544    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896547    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896550    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896552    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896555    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896557    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896560    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896564    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896566    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896569    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896572    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:04.909954 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896575    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896591    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896595    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896598    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896601    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896605    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896608    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896611    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896613    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896616    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896619    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896623    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896625    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896628    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.896630    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.904965    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.904983    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.905004    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.905012    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:36:04.910442 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905098    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905106    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905112    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905118    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905122    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905127    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905131    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905135    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905139    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905143    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905147    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905152    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905158    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905162    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905168    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905172    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905177    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905182    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905186    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905190    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:04.910933 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905194    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905198    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905203    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905207    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905212    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905216    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905223    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905230    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905235    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905240    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905245    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905250    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905254    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905258    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905262    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905267    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905271    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905275    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905279    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:04.911464 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905283    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905288    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905292    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905297    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905301    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905305    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905311    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905315    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905319    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905324    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905328    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905334    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905338    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905343    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905347    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905352    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905357    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905361    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905365    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:04.911995 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905369    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905373    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905378    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905383    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905387    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905391    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905396    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905400    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905404    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905408    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905414    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905420    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905425    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905430    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905435    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905439    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905444    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905448    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905452    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905456    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:04.912448 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905462    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905466    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905471    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905475    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905479    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905483    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905487    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905492    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905495    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905500    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905505    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905509    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905514    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905518    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905522    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905527    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905531    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905535    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905540    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905544    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:04.913077 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905549    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905553    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905557    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905562    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905566    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905570    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905575    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905598    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.905640    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905869    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905877    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905882    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905887    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905892    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905896    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905901    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:04.913781 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905905    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905910    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905914    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905918    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905923    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905927    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905932    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905936    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905941    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905945    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905949    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905954    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905958    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905962    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905967    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905971    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905975    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905979    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905983    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905987    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:04.914172 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905991    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905995    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.905998    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906002    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906006    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906011    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906015    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906019    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906023    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906028    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906033    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906037    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906041    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906045    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906050    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906054    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906058    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906062    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906066    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:04.914936 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906073    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906080    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906086    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906090    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906095    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906099    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906104    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906108    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906112    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906117    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906121    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906125    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906129    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906133    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906137    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906142    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906146    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906150    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906155    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:04.915636 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906159    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906164    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906168    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906172    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906176    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906182    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906186    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906190    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906195    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906199    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906203    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906208    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906212    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906217    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906221    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906225    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906229    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906233    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906238    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906242    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:04.916350 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906246    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906250    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906255    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906259    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906263    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906267    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906272    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906276    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906280    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906284    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906288    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906292    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906297    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906301    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906305    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906312    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906318    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906323    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906329    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:04.917244 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906334    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906338    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.906378    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906449    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906456    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906461    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906465    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906470    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906474    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906478    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906483    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906487    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906491    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906495    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906500    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906504    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:04.917775 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906509    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906516    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906520    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906524    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906529    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906533    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906538    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906542    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906546    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906550    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906554    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906558    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906563    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906567    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906572    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906593    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906599    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906604    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906608    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906612    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:04.918169 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906616    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906620    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906624    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906628    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906635    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906640    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906645    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906649    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906654    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906658    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906662    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906666    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906671    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906678    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906683    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906687    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906691    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906696    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906700    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:04.918650 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906704    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906708    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906712    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906717    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906722    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906726    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906730    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906735    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906739    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906744    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906749    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906753    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906757    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906761    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906765    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906770    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906774    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906778    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906782    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906787    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:04.919075 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906791    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906795    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906799    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906803    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906808    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906812    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906818    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906822    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906827    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906831    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906835    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906839    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906845    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906850    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906854    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906858    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906862    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906869    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906875    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:04.919547 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906879    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906883    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906888    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906892    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906897    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906901    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906905    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906909    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906914    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906918    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906922    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906927    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906931    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906935    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:04.906939    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.907020    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.909852    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.911071    2571 server.go:1014] "Starting client certificate rotation"
Oct 30 18:36:04.920008 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.911165    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:04.920428 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.912108    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:04.920428 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.913404    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:04.920428 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.918029    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:04.930772 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.930755    2571 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:36:04.934425 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.934410    2571 log.go:25] "Validated CRI v1 image API"
Oct 30 18:36:04.935268 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.935237    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:36:04.939548 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.939527    2571 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8f5b8466-d342-4f5d-b3a4-1c2358c3bf9b:/dev/nvme0n1p3 97db2337-79f5-4adf-8f0c-2fabe5b9d224:/dev/nvme0n1p4]
Oct 30 18:36:04.939607 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.939548    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:36:04.944383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.944278    2571 manager.go:217] Machine: {Timestamp:2026-10-30 18:36:04.943817327 +0000 UTC m=+0.388615089 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100995 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2610c0e0988a0f9bcca89acb9dc73c SystemUUID:ec2610c0-e098-8a0f-9bcc-a89acb9dc73c BootID:c87c4750-4d78-4fc5-8953-afd9eaafc18c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d3:e4:1b:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d3:e4:1b:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:81:3c:0d:03:8a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:36:04.945263 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.945252    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:36:04.945384 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.945368    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:36:04.949053 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.949027    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:36:04.949195 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.949055    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-92.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:36:04.949237 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.949198    2571 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:36:04.949237 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.949206    2571 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:36:04.949237 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.949229    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:04.949315 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.949244    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:04.950318 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.950307    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:36:04.953941 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.953923    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:36:04.954048 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.954039    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:36:04.954896 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.954886    2571 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:36:04.954945 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.954899    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:36:04.954945 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.954914    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:36:04.954945 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.954923    2571 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:36:04.954945 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.954934    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:36:04.955261 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.955250    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:04.957661 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.957645    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:36:04.963623 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.963607    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:04.965085 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965072    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:36:04.965155 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965102    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:36:04.965265 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965251    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:36:04.965306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965274    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:36:04.965306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965285    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:36:04.965306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965295    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:36:04.965306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965302    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:36:04.965456 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965312    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:36:04.965962 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965951    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:36:04.965962 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965961    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:36:04.966039 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965969    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:36:04.966039 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965975    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:36:04.966039 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965988    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:36:04.966039 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.965997    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:36:04.966971 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.966961    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:36:04.966971 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.966970    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:36:04.970546 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.970533    2571 server.go:1263] "Started kubelet"
Oct 30 18:36:04.971867 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.971816    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:36:04.972093 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.972036    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:36:04.972175 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.972163    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:36:04.972648 ip-10-0-128-92 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:36:04.974010 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.973764    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:36:04.974951 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.974930    2571 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:36:04.980059 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:04.980034    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:36:04.983881 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.983859    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:04.985068 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.985053    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:36:04.985173 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:04.985156    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-92.us-west-2.compute.internal\" not found"
Oct 30 18:36:04.985974 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.985958    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:36:04.985974 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.985975    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:36:04.986114 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.986103    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:36:04.986154 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.986142    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:36:04.986294 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.986157    2571 factory.go:55] Registering systemd factory
Oct 30 18:36:04.986294 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.986167    2571 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:36:04.986294 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.986214    2571 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:36:04.986294 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.986226    2571 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:36:04.986474 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.986394    2571 factory.go:153] Registering CRI-O factory
Oct 30 18:36:04.986474 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.986408    2571 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:36:04.986474 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.986427    2571 factory.go:103] Registering Raw factory
Oct 30 18:36:04.986474 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.986451    2571 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:36:04.987462 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.987449    2571 manager.go:319] Starting recovery of all containers
Oct 30 18:36:04.995541 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:04.995525    2571 manager.go:324] Recovery completed
Oct 30 18:36:05.000453 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.000440    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:05.003121 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.003103    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:05.003180 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.003145    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:05.003180 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.003161    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:05.003779 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.003767    2571 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:36:05.003779 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.003777    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:36:05.003896 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.003794    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:36:05.003896 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.003874    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:05.004644 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.004628    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:05.004728 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.004661    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:05.004728 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.004676    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:05.005276 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.005264    2571 policy_none.go:50] "Start"
Oct 30 18:36:05.005339 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.005280    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:36:05.005339 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.005290    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:36:05.006867 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.006856    2571 policy_none.go:44] "Start"
Oct 30 18:36:05.045169 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.045138    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:36:05.047558 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.047547    2571 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:36:05.047657 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:05.047595    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:36:05.047657 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.047608    2571 server.go:87] "Starting device plugin registration server"
Oct 30 18:36:05.047883 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.047870    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:36:05.047948 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.047884    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:36:05.048055 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.048035    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:36:05.048163 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.048137    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:36:05.048163 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.048146    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:36:05.048706 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:05.048685    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:36:05.048818 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:05.048738    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-92.us-west-2.compute.internal\" not found"
Oct 30 18:36:05.106344 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:05.106314    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-92.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:36:05.107364 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:05.106316    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-92.us-west-2.compute.internal.18e3627cd87389a6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-92.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-92.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:04.970506662 +0000 UTC m=+0.415304420,LastTimestamp:2026-10-30 18:36:04.970506662 +0000 UTC m=+0.415304420,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-92.us-west-2.compute.internal,}"
Oct 30 18:36:05.148713 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.148646    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:05.149600 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.149568    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:05.149688 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.149619    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:05.149688 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.149635    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:05.149688 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.149668    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.152251 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.152236    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-92.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:36:05.198997 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.198835    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:36:05.200274 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.198933    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hhpjt"
Oct 30 18:36:05.211698 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.211670    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:36:05.212920 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.212900    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:36:05.212992 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.212926    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:36:05.212992 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.212952    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:36:05.213077 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:05.212991    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:36:05.265003 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.264976    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:05.266849 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.266835    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:05.313851 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.313825    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"]
Oct 30 18:36:05.313959 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.313880    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:05.314857 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.314842    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:05.314934 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.314871    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:05.314934 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.314882    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:05.316474 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.316462    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:05.316636 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.316622    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.316675 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.316652    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:05.317138 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.317126    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:05.317196 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.317154    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:05.317196 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.317167    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:05.317196 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.317131    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:05.317305 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.317210    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:05.317305 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.317226    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:05.318593 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.318560    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.318685 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.318604    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:05.319249 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.319235    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:05.319322 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.319257    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:05.319322 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.319271    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:05.348499 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:05.348474    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-92.us-west-2.compute.internal\" not found" node="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.354925 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:05.354908    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-92.us-west-2.compute.internal\" not found" node="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.365701 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.365681    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:05.367257 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.367242    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:05.368053 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.368038    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.368124 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:05.368063    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-92.us-west-2.compute.internal\": node \"ip-10-0-128-92.us-west-2.compute.internal\" not found"
Oct 30 18:36:05.385930 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.385904    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.388107 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.388087    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ce1444a469ee2b4c13e78a90a9b0681-config\") pod \"kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"4ce1444a469ee2b4c13e78a90a9b0681\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.388165 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.388115    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.388165 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.388132    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.444601 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.444525    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:05.445648 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.445636    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.489028 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.489005    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ce1444a469ee2b4c13e78a90a9b0681-config\") pod \"kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"4ce1444a469ee2b4c13e78a90a9b0681\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.489135 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.489033    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.489135 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.489050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.489135 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.489112    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.489135 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.489123    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ce1444a469ee2b4c13e78a90a9b0681-config\") pod \"kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"4ce1444a469ee2b4c13e78a90a9b0681\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.489258 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.489172    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.505362 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.505342    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:05.657171 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.657142    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.659077 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.659057    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:05.911035 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.910959    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:36:05.911690 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.911064    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:36:05.911690 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.911095    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:36:05.911690 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.911130    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:36:05.911690 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:05.911156    2571 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a511c06f6c4c24b5eb9148195e6ce288-46029cde054419ce.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-92.us-west-2.compute.internal.18e3627cda65e7ef\": read tcp 10.0.128.92:33130->18.189.88.140:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-92.us-west-2.compute.internal.18e3627cda65e7ef  default   5208 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-92.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-92.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-92.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:05 +0000 UTC,LastTimestamp:2026-10-30 18:36:05.149641456 +0000 UTC m=+0.594439216,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-92.us-west-2.compute.internal,}"
Oct 30 18:36:05.955284 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.955264    2571 apiserver.go:52] "Watching apiserver"
Oct 30 18:36:05.985616 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:05.985596    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:06.203781 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.203693    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:31:05 +0000 UTC" deadline="2028-07-22 06:41:16.755509212 +0000 UTC"
Oct 30 18:36:06.203781 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.203723    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15132h5m10.551790215s"
Oct 30 18:36:06.219790 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.219770    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:36:06.220120 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.220099    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-84s2b","openshift-multus/multus-vwxwq","openshift-network-operator/iptables-alerter-5xvkc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns","openshift-dns/node-resolver-sw4vz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-5phnp","kube-system/global-pull-secret-syncer-55w4h","kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-rrhvc","openshift-image-registry/node-ca-9pbdd","openshift-multus/network-metrics-daemon-4zt8w","openshift-ovn-kubernetes/ovnkube-node-nddlj","kube-system/konnectivity-agent-rqbk8"]
Oct 30 18:36:06.221886 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.221864    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.223284 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.223261    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.224674 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.224641    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.226074 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.226053    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.227547 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.227532    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.228974 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.228955    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:06.229071 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.229011    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:06.230521 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.230496    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:06.230607 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.230569    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:06.231993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.231974    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.233541 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.233524    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.234979 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.234954    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:06.235126 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.235013    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:06.236676 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.236656    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.238340 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.238320    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:06.275669 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.275635    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:36:06.275862 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.275838    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:36:06.275946 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.275883    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:36:06.275946 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.275890    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h4btl\""
Oct 30 18:36:06.275946 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.275890    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:36:06.275946 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.275914    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:36:06.276090 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.275998    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5mz5g\""
Oct 30 18:36:06.276090 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.276035    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:36:06.278359 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.278344    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:06.278423 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.278416    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:06.278469 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.278432    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:36:06.278652 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.278635    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:36:06.278652 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.278663    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j6hdp\""
Oct 30 18:36:06.278778 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.278706    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:36:06.278778 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.278714    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:36:06.278844 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.278790    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dnpxx\""
Oct 30 18:36:06.279889 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.279873    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:36:06.279946 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.279904    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vpn2j\""
Oct 30 18:36:06.280188 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.280170    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:36:06.287961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.287945    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:36:06.292713 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292691    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.292805 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-systemd\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.292805 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292733    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-config\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.292805 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292748    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c50a391f-9c33-4ece-8c52-e767af7a41b1-host-slash\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.292805 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292777    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.292939 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292845    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jhk6\" (UniqueName: \"kubernetes.io/projected/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-kube-api-access-7jhk6\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.292939 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292918    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.292998 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292936    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6bj4\" (UniqueName: \"kubernetes.io/projected/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-kube-api-access-n6bj4\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.292998 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-slash\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.292998 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292967    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-node-log\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.292998 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.292991    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-device-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.293124 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293018    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:06.293124 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293046    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-socket-dir-parent\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.293124 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-run\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.293124 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293100    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-sys\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.293243 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293146    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-systemd-units\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.293243 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293177    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.293243 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293196    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-sys-fs\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.293243 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293211    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp9pl\" (UniqueName: \"kubernetes.io/projected/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-kube-api-access-vp9pl\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:06.293243 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293225    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-etc-kubernetes\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.293243 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293240    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:06.293434 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293255    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysconfig\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.293434 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293269    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-host\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.293434 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293301    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-etc-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.293434 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293315    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-script-lib\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.293434 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293331    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-tmp-dir\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.293434 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-tuned\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.293434 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293374    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-systemd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.293434 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293396    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2nlc\" (UniqueName: \"kubernetes.io/projected/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-kube-api-access-f2nlc\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.293434 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293412    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-conf\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293452    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293483    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-k8s-cni-cncf-io\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293508    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-netns\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293551    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-multus\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-host\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-serviceca\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293665    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-log-socket\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293717    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-system-cni-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293737    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-kubelet\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-netd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cnibin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.293826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qxp\" (UniqueName: \"kubernetes.io/projected/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-kube-api-access-z6qxp\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293838    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-netns\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293860    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-system-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293879    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-daemon-config\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-hosts-file\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293919    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-lib-modules\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293940    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-var-lib-kubelet\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.293971    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovn-node-metrics-cert\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294006    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-cnibin\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mltx7\" (UniqueName: \"kubernetes.io/projected/65831eb9-258f-4f08-96b0-1b4556b183a7-kube-api-access-mltx7\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294041    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-multus-certs\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294063    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-kubelet-config\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294080    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-var-lib-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294103    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-registration-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294176    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cni-binary-copy\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.294321 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-dbus\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294244    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294270    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp648\" (UniqueName: \"kubernetes.io/projected/c50a391f-9c33-4ece-8c52-e767af7a41b1-kube-api-access-xp648\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-modprobe-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294359    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-tmp\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294381    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-bin\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294400    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c50a391f-9c33-4ece-8c52-e767af7a41b1-iptables-alerter-script\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294427    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-socket-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294458    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294479    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92v5\" (UniqueName: \"kubernetes.io/projected/12185714-c5ea-442a-9f5d-ef7af20f28c0-kube-api-access-t92v5\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294503    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-agent-certs\") pod \"konnectivity-agent-rqbk8\" (UID: \"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9\") " pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-hostroot\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294546    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-conf-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-kubernetes\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294623    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-ovn\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.294828 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294644    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-env-overrides\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.295283 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r7z9\" (UniqueName: \"kubernetes.io/projected/6350fe6c-a551-41dc-88a2-f15bd2360a1c-kube-api-access-4r7z9\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.295283 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294699    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-konnectivity-ca\") pod \"konnectivity-agent-rqbk8\" (UID: \"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9\") " pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:06.295283 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294718    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-bin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.295283 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-kubelet\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.295283 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294773    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.295283 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.294792    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.323028 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.323003    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:06.323110 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.323060    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:06.323151 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.323109    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vd777\""
Oct 30 18:36:06.323151 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.323141    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m6qzp\""
Oct 30 18:36:06.323378 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.323365    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:36:06.323412 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.323387    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:36:06.323690 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.323674    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:36:06.382706 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.382681    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:36:06.382806 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.382724    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t795z"
Oct 30 18:36:06.384447 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.384421    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:36:06.384809 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.384561    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:36:06.384809 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.384661    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:36:06.384809 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.384704    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7kzh2\""
Oct 30 18:36:06.384809 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.384775    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:36:06.385018 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.384873    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:36:06.385301 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.385279    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:36:06.385375 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.385339    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlfn7\""
Oct 30 18:36:06.387064 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.385525    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:36:06.387064 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.385971    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:36:06.395236 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395215    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.395350 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp648\" (UniqueName: \"kubernetes.io/projected/c50a391f-9c33-4ece-8c52-e767af7a41b1-kube-api-access-xp648\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.395350 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395280    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-modprobe-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.395350 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395305    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-tmp\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.395350 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395322    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.395350 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395329    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-bin\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c50a391f-9c33-4ece-8c52-e767af7a41b1-iptables-alerter-script\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395377    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-socket-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92v5\" (UniqueName: \"kubernetes.io/projected/12185714-c5ea-442a-9f5d-ef7af20f28c0-kube-api-access-t92v5\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395445    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-modprobe-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395463    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-agent-certs\") pod \"konnectivity-agent-rqbk8\" (UID: \"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9\") " pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395488    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-hostroot\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-conf-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395539    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-kubernetes\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-ovn\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-env-overrides\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.395642 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395613    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7z9\" (UniqueName: \"kubernetes.io/projected/6350fe6c-a551-41dc-88a2-f15bd2360a1c-kube-api-access-4r7z9\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-konnectivity-ca\") pod \"konnectivity-agent-rqbk8\" (UID: \"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9\") " pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395382    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-bin\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395668    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395678    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-bin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395707    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-kubelet\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395716    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-kubernetes\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395726    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-conf-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395734    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395739    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-hostroot\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395759    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-socket-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395781    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395809    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-bin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395831    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-ovn\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-systemd\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395900    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395925    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-config\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.396383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395889    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-systemd\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395957    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-kubelet\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.395992    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c50a391f-9c33-4ece-8c52-e767af7a41b1-iptables-alerter-script\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396014    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396026    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396069    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c50a391f-9c33-4ece-8c52-e767af7a41b1-host-slash\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396107    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396137    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jhk6\" (UniqueName: \"kubernetes.io/projected/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-kube-api-access-7jhk6\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396165    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396183    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c50a391f-9c33-4ece-8c52-e767af7a41b1-host-slash\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396192    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6bj4\" (UniqueName: \"kubernetes.io/projected/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-kube-api-access-n6bj4\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-slash\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396278    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-node-log\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396298    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-env-overrides\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396319    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-device-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396360    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-slash\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396374    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-konnectivity-ca\") pod \"konnectivity-agent-rqbk8\" (UID: \"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9\") " pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:06.397348 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396378    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-socket-dir-parent\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-run\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396458    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-sys\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396468    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-node-log\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396434    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-device-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396511    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-systemd-units\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396521    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396539    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396565    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-config\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396569    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-systemd-units\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396590    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-sys\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396604    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-sys-fs\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396639    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396657    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9pl\" (UniqueName: \"kubernetes.io/projected/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-kube-api-access-vp9pl\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396673    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-sys-fs\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396687    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-etc-kubernetes\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396713    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:06.398256 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396731    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysconfig\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396746    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-host\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396774    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-etc-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396792    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-script-lib\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396816    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysconfig\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.396821    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396834    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-tmp-dir\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396851    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-socket-dir-parent\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396859    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-tuned\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396861    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-etc-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.396892    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret podName:2601c6f1-0eb6-4038-b54c-5f8a7214db42 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:06.896873801 +0000 UTC m=+2.341671565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret") pod "global-pull-secret-syncer-55w4h" (UID: "2601c6f1-0eb6-4038-b54c-5f8a7214db42") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396909    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-run\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396913    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-systemd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396922    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-host\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396962    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2nlc\" (UniqueName: \"kubernetes.io/projected/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-kube-api-access-f2nlc\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396978    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-systemd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.398993 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396976    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-etc-kubernetes\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.396961    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-conf\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397027    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397054    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-k8s-cni-cncf-io\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397082    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-netns\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397109    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-multus\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397116    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397132    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-host\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397166    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-serviceca\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397228    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-conf\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397229    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-netns\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397252    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-log-socket\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397256    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-tmp-dir\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397275    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-multus\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397300    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-log-socket\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397311    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-host\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397324    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-system-cni-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.399621 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397344    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-k8s-cni-cncf-io\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397355    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397379    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-system-cni-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-kubelet\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-netd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397434    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cnibin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397436    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-script-lib\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397458    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qxp\" (UniqueName: \"kubernetes.io/projected/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-kube-api-access-z6qxp\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-netns\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397493    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-netd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397510    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-system-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397517    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cnibin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397534    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-kubelet\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397536    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-daemon-config\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-hosts-file\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397593    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-serviceca\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397598    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-lib-modules\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397623    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-system-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.400306 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397672    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-var-lib-kubelet\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397698    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-lib-modules\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovn-node-metrics-cert\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397741    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-cnibin\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltx7\" (UniqueName: \"kubernetes.io/projected/65831eb9-258f-4f08-96b0-1b4556b183a7-kube-api-access-mltx7\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397797    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-multus-certs\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397818    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-kubelet-config\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-var-lib-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397862    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397879    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-registration-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397895    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cni-binary-copy\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-dbus\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397939    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-var-lib-kubelet\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.397945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.398029    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-var-lib-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.398084    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-hosts-file\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.398125    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-cnibin\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.400850 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.398367    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-multus-certs\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.398378    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-netns\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.398425    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-kubelet-config\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.398489    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-dbus\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.398502    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.398540    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.398594    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.398599    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-registration-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.398641    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs podName:cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:06.898625275 +0000 UTC m=+2.343423036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs") pod "network-metrics-daemon-4zt8w" (UID: "cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.399329    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-daemon-config\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.399508    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cni-binary-copy\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.400081    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-tuned\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.400150    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-tmp\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.401367 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.400385    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-agent-certs\") pod \"konnectivity-agent-rqbk8\" (UID: \"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9\") " pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:06.401956 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.401622    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovn-node-metrics-cert\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.454146 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.454090    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:06.454146 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.454142    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:06.454305 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.454171    2571 projected.go:196] Error preparing data for projected volume kube-api-access-4hpcs for pod openshift-network-diagnostics/network-check-target-5phnp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:06.454305 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.454246    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs podName:92bb0644-0b77-405a-bcf2-4b88b8ac4d4d nodeName:}" failed. No retries permitted until 2026-10-30 18:36:06.954226571 +0000 UTC m=+2.399024332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4hpcs" (UniqueName: "kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs") pod "network-check-target-5phnp" (UID: "92bb0644-0b77-405a-bcf2-4b88b8ac4d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:06.461079 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.461047    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92v5\" (UniqueName: \"kubernetes.io/projected/12185714-c5ea-442a-9f5d-ef7af20f28c0-kube-api-access-t92v5\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.461199 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.461126    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp648\" (UniqueName: \"kubernetes.io/projected/c50a391f-9c33-4ece-8c52-e767af7a41b1-kube-api-access-xp648\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.461266 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.461215    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6bj4\" (UniqueName: \"kubernetes.io/projected/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-kube-api-access-n6bj4\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.461503 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.461478    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltx7\" (UniqueName: \"kubernetes.io/projected/65831eb9-258f-4f08-96b0-1b4556b183a7-kube-api-access-mltx7\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.461580 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.461513    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7z9\" (UniqueName: \"kubernetes.io/projected/6350fe6c-a551-41dc-88a2-f15bd2360a1c-kube-api-access-4r7z9\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.461849 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.461831    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2nlc\" (UniqueName: \"kubernetes.io/projected/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-kube-api-access-f2nlc\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.462983 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.462962    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qxp\" (UniqueName: \"kubernetes.io/projected/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-kube-api-access-z6qxp\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.463449 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.463425    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jhk6\" (UniqueName: \"kubernetes.io/projected/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-kube-api-access-7jhk6\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.463503 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.463460    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9pl\" (UniqueName: \"kubernetes.io/projected/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-kube-api-access-vp9pl\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:06.563971 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.563935    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:06.573293 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.573272    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:06.587423 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.587400    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:06.610569 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.610549    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:06.610742 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.610724    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:06.622961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.622941    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:06.635506 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.635488    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:06.648694 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.648666    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:06.652702 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.652684    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:06.752076 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:06.752035    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b49969aba00a9d5dd95ce58685c160.slice/crio-7671310b8bd536d1490c6752b1cfbd4d4a092141a5d767a83f57e6c0a7b0d4ee WatchSource:0}: Error finding container 7671310b8bd536d1490c6752b1cfbd4d4a092141a5d767a83f57e6c0a7b0d4ee: Status 404 returned error can't find the container with id 7671310b8bd536d1490c6752b1cfbd4d4a092141a5d767a83f57e6c0a7b0d4ee
Oct 30 18:36:06.759052 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.759036    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:36:06.780444 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:06.780417    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce1444a469ee2b4c13e78a90a9b0681.slice/crio-831b762ae9cc950299f132beb1715e116a8afc93faa442c3c7f8ea4eed440d64 WatchSource:0}: Error finding container 831b762ae9cc950299f132beb1715e116a8afc93faa442c3c7f8ea4eed440d64: Status 404 returned error can't find the container with id 831b762ae9cc950299f132beb1715e116a8afc93faa442c3c7f8ea4eed440d64
Oct 30 18:36:06.902243 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.902206    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:06.902412 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:06.902267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:06.902412 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.902376    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:06.902484 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.902437    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs podName:cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:07.902423829 +0000 UTC m=+3.347221600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs") pod "network-metrics-daemon-4zt8w" (UID: "cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:06.902484 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.902376    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:06.902484 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:06.902474    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret podName:2601c6f1-0eb6-4038-b54c-5f8a7214db42 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:07.902467881 +0000 UTC m=+3.347265676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret") pod "global-pull-secret-syncer-55w4h" (UID: "2601c6f1-0eb6-4038-b54c-5f8a7214db42") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:07.003446 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:07.003362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:07.003996 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:07.003500    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:07.003996 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:07.003515    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:07.003996 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:07.003524    2571 projected.go:196] Error preparing data for projected volume kube-api-access-4hpcs for pod openshift-network-diagnostics/network-check-target-5phnp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:07.003996 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:07.003569    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs podName:92bb0644-0b77-405a-bcf2-4b88b8ac4d4d nodeName:}" failed. No retries permitted until 2026-10-30 18:36:08.003557787 +0000 UTC m=+3.448355533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hpcs" (UniqueName: "kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs") pod "network-check-target-5phnp" (UID: "92bb0644-0b77-405a-bcf2-4b88b8ac4d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:07.086791 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:07.086762    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:07.111275 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:07.111247    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:07.216921 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:07.216864    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal" event={"ID":"4ce1444a469ee2b4c13e78a90a9b0681","Type":"ContainerStarted","Data":"831b762ae9cc950299f132beb1715e116a8afc93faa442c3c7f8ea4eed440d64"}
Oct 30 18:36:07.217945 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:07.217921    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal" event={"ID":"e4b49969aba00a9d5dd95ce58685c160","Type":"ContainerStarted","Data":"7671310b8bd536d1490c6752b1cfbd4d4a092141a5d767a83f57e6c0a7b0d4ee"}
Oct 30 18:36:07.353224 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:07.353136    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:07.386498 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:07.386464    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:31:06 +0000 UTC" deadline="2028-05-21 12:58:31.416675259 +0000 UTC"
Oct 30 18:36:07.386498 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:07.386495    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13650h22m24.030183859s"
Oct 30 18:36:07.433081 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:07.433042    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50a391f_9c33_4ece_8c52_e767af7a41b1.slice/crio-7b70d86fbf99aa801145917ef02735a548b46e2c2eb41d357f0722007c48e301 WatchSource:0}: Error finding container 7b70d86fbf99aa801145917ef02735a548b46e2c2eb41d357f0722007c48e301: Status 404 returned error can't find the container with id 7b70d86fbf99aa801145917ef02735a548b46e2c2eb41d357f0722007c48e301
Oct 30 18:36:07.457812 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:07.457780    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d6f30f_5f4f_4b67_9dcc_3cae33390c84.slice/crio-abe7d5a8d94625e7d5425c441f0d6f029510612afca0199219d74290fac515ec WatchSource:0}: Error finding container abe7d5a8d94625e7d5425c441f0d6f029510612afca0199219d74290fac515ec: Status 404 returned error can't find the container with id abe7d5a8d94625e7d5425c441f0d6f029510612afca0199219d74290fac515ec
Oct 30 18:36:07.474630 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:07.474606    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb976d8d0_4b1e_43a8_874d_94d3ecde6a3c.slice/crio-1bb931b9e061af7cd61facf138ea815b12ee4acec95ca94560c27f9666c09476 WatchSource:0}: Error finding container 1bb931b9e061af7cd61facf138ea815b12ee4acec95ca94560c27f9666c09476: Status 404 returned error can't find the container with id 1bb931b9e061af7cd61facf138ea815b12ee4acec95ca94560c27f9666c09476
Oct 30 18:36:07.487030 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:07.486971    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65831eb9_258f_4f08_96b0_1b4556b183a7.slice/crio-68cadacbe7916641daa1f4eec003e4ed0cf78fc56c3a12501e8d13700c88dba8 WatchSource:0}: Error finding container 68cadacbe7916641daa1f4eec003e4ed0cf78fc56c3a12501e8d13700c88dba8: Status 404 returned error can't find the container with id 68cadacbe7916641daa1f4eec003e4ed0cf78fc56c3a12501e8d13700c88dba8
Oct 30 18:36:07.534613 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:07.534581    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6350fe6c_a551_41dc_88a2_f15bd2360a1c.slice/crio-4406b0a0d4aaf7f96fcde8da9366d4fd03593a4643176dc9800015a852daabe5 WatchSource:0}: Error finding container 4406b0a0d4aaf7f96fcde8da9366d4fd03593a4643176dc9800015a852daabe5: Status 404 returned error can't find the container with id 4406b0a0d4aaf7f96fcde8da9366d4fd03593a4643176dc9800015a852daabe5
Oct 30 18:36:07.571351 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:07.571322    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c6cdf4_e8d7_4a93_9c44_98fdf90250b5.slice/crio-ab41a0186969c867e8f840dba647ab276dc9a1f4d8a0a2f1504d68f29ad786d1 WatchSource:0}: Error finding container ab41a0186969c867e8f840dba647ab276dc9a1f4d8a0a2f1504d68f29ad786d1: Status 404 returned error can't find the container with id ab41a0186969c867e8f840dba647ab276dc9a1f4d8a0a2f1504d68f29ad786d1
Oct 30 18:36:07.581606 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:07.581578    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12185714_c5ea_442a_9f5d_ef7af20f28c0.slice/crio-1115cd671725385db8c19e02126d4a56396798424abc30ad0bbc5c2eae3c2e63 WatchSource:0}: Error finding container 1115cd671725385db8c19e02126d4a56396798424abc30ad0bbc5c2eae3c2e63: Status 404 returned error can't find the container with id 1115cd671725385db8c19e02126d4a56396798424abc30ad0bbc5c2eae3c2e63
Oct 30 18:36:07.584240 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:07.584210    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a4e656_ef45_4ef7_b2f5_bbb46c7fc600.slice/crio-3eeb1c8d921c3819becb1e00c18a85617da55d8965045cfbce0e368a84e1bd8f WatchSource:0}: Error finding container 3eeb1c8d921c3819becb1e00c18a85617da55d8965045cfbce0e368a84e1bd8f: Status 404 returned error can't find the container with id 3eeb1c8d921c3819becb1e00c18a85617da55d8965045cfbce0e368a84e1bd8f
Oct 30 18:36:07.586313 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:07.586289    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57df4ae_cbc2_41ac_ab43_c7c21cbe65d9.slice/crio-359d26e716db916f5470c236c86b93b32950f8aa1b8d36310d295bd2fcfc50f2 WatchSource:0}: Error finding container 359d26e716db916f5470c236c86b93b32950f8aa1b8d36310d295bd2fcfc50f2: Status 404 returned error can't find the container with id 359d26e716db916f5470c236c86b93b32950f8aa1b8d36310d295bd2fcfc50f2
Oct 30 18:36:07.911569 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:07.911526    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:07.911776 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:07.911595    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:07.911776 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:07.911771    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:07.911910 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:07.911838    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs podName:cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:09.911818552 +0000 UTC m=+5.356616317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs") pod "network-metrics-daemon-4zt8w" (UID: "cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:07.911910 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:07.911901    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:07.912026 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:07.911926    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret podName:2601c6f1-0eb6-4038-b54c-5f8a7214db42 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:09.911918198 +0000 UTC m=+5.356715958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret") pod "global-pull-secret-syncer-55w4h" (UID: "2601c6f1-0eb6-4038-b54c-5f8a7214db42") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:08.012770 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.012728    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:08.013257 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:08.012906    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:08.013257 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:08.012930    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:08.013257 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:08.012940    2571 projected.go:196] Error preparing data for projected volume kube-api-access-4hpcs for pod openshift-network-diagnostics/network-check-target-5phnp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:08.013257 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:08.012994    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs podName:92bb0644-0b77-405a-bcf2-4b88b8ac4d4d nodeName:}" failed. No retries permitted until 2026-10-30 18:36:10.012979476 +0000 UTC m=+5.457777240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hpcs" (UniqueName: "kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs") pod "network-check-target-5phnp" (UID: "92bb0644-0b77-405a-bcf2-4b88b8ac4d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:08.214023 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.213896    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:08.214229 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.213896    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:08.214229 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:08.214054    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:08.214229 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.214080    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:08.214229 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:08.214201    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:08.214585 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:08.214279    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:08.221903 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.221871    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9pbdd" event={"ID":"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600","Type":"ContainerStarted","Data":"3eeb1c8d921c3819becb1e00c18a85617da55d8965045cfbce0e368a84e1bd8f"}
Oct 30 18:36:08.223502 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.223458    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns" event={"ID":"12185714-c5ea-442a-9f5d-ef7af20f28c0","Type":"ContainerStarted","Data":"1115cd671725385db8c19e02126d4a56396798424abc30ad0bbc5c2eae3c2e63"}
Oct 30 18:36:08.224934 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.224913    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84s2b" event={"ID":"65831eb9-258f-4f08-96b0-1b4556b183a7","Type":"ContainerStarted","Data":"68cadacbe7916641daa1f4eec003e4ed0cf78fc56c3a12501e8d13700c88dba8"}
Oct 30 18:36:08.226349 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.226319    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vwxwq" event={"ID":"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c","Type":"ContainerStarted","Data":"1bb931b9e061af7cd61facf138ea815b12ee4acec95ca94560c27f9666c09476"}
Oct 30 18:36:08.227695 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.227670    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5xvkc" event={"ID":"c50a391f-9c33-4ece-8c52-e767af7a41b1","Type":"ContainerStarted","Data":"7b70d86fbf99aa801145917ef02735a548b46e2c2eb41d357f0722007c48e301"}
Oct 30 18:36:08.229123 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.229099    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rqbk8" event={"ID":"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9","Type":"ContainerStarted","Data":"359d26e716db916f5470c236c86b93b32950f8aa1b8d36310d295bd2fcfc50f2"}
Oct 30 18:36:08.230880 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.230854    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rrhvc" event={"ID":"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5","Type":"ContainerStarted","Data":"ab41a0186969c867e8f840dba647ab276dc9a1f4d8a0a2f1504d68f29ad786d1"}
Oct 30 18:36:08.232314 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.232285    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj" event={"ID":"6350fe6c-a551-41dc-88a2-f15bd2360a1c","Type":"ContainerStarted","Data":"4406b0a0d4aaf7f96fcde8da9366d4fd03593a4643176dc9800015a852daabe5"}
Oct 30 18:36:08.234195 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.234170    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sw4vz" event={"ID":"07d6f30f-5f4f-4b67-9dcc-3cae33390c84","Type":"ContainerStarted","Data":"abe7d5a8d94625e7d5425c441f0d6f029510612afca0199219d74290fac515ec"}
Oct 30 18:36:08.386802 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.386758    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:31:06 +0000 UTC" deadline="2028-07-14 21:22:52.443392764 +0000 UTC"
Oct 30 18:36:08.386802 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:08.386800    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14954h46m44.056598841s"
Oct 30 18:36:09.933295 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:09.932795    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:09.933295 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:09.932877    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:09.933295 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:09.933050    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:09.933295 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:09.933116    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs podName:cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:13.933097099 +0000 UTC m=+9.377894881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs") pod "network-metrics-daemon-4zt8w" (UID: "cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:09.933295 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:09.933179    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:09.933295 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:09.933212    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret podName:2601c6f1-0eb6-4038-b54c-5f8a7214db42 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:13.933201746 +0000 UTC m=+9.377999498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret") pod "global-pull-secret-syncer-55w4h" (UID: "2601c6f1-0eb6-4038-b54c-5f8a7214db42") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:10.033711 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:10.033314    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:10.033711 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:10.033478    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:10.033711 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:10.033503    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:10.033711 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:10.033517    2571 projected.go:196] Error preparing data for projected volume kube-api-access-4hpcs for pod openshift-network-diagnostics/network-check-target-5phnp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:10.033711 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:10.033574    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs podName:92bb0644-0b77-405a-bcf2-4b88b8ac4d4d nodeName:}" failed. No retries permitted until 2026-10-30 18:36:14.033556981 +0000 UTC m=+9.478354732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hpcs" (UniqueName: "kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs") pod "network-check-target-5phnp" (UID: "92bb0644-0b77-405a-bcf2-4b88b8ac4d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:10.213857 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:10.213259    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:10.213857 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:10.213403    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:10.213857 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:10.213488    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:10.213857 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:10.213564    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:10.213857 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:10.213644    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:10.213857 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:10.213758    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:11.308368 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:11.307401    2571 generic.go:356] "Generic (PLEG): container finished" podID="e4b49969aba00a9d5dd95ce58685c160" containerID="5a3e22e86f441d0d9181473e64dec9ecfcc223f78239b07658881d878a07bc4a" exitCode=0
Oct 30 18:36:11.308368 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:11.307738    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal" event={"ID":"e4b49969aba00a9d5dd95ce58685c160","Type":"ContainerDied","Data":"5a3e22e86f441d0d9181473e64dec9ecfcc223f78239b07658881d878a07bc4a"}
Oct 30 18:36:12.213504 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:12.213464    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:12.213729 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:12.213643    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:12.213729 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:12.213716    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:12.213850 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:12.213795    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:12.214260 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:12.214238    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:12.214443 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:12.214416    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:12.314641 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:12.314586    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal" event={"ID":"e4b49969aba00a9d5dd95ce58685c160","Type":"ContainerStarted","Data":"3a478f7c6c1683e7a623b04a9caa25059d28d57de9e116a39cf6ce3a69df961c"}
Oct 30 18:36:12.318613 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:12.318581    2571 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="fdd9fe55abff8e5ff91754dd02a89906836215bb4aebc866f88c712768c90086" exitCode=0
Oct 30 18:36:12.318769 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:12.318692    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84s2b" event={"ID":"65831eb9-258f-4f08-96b0-1b4556b183a7","Type":"ContainerDied","Data":"fdd9fe55abff8e5ff91754dd02a89906836215bb4aebc866f88c712768c90086"}
Oct 30 18:36:12.322853 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:12.322828    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal" event={"ID":"4ce1444a469ee2b4c13e78a90a9b0681","Type":"ContainerStarted","Data":"f1b6da25bdff6522a7ff64ea65d5b4f4f77427d58e4b0140d1495baf5771019d"}
Oct 30 18:36:12.530563 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:12.530446    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal" podStartSLOduration=7.530423621 podStartE2EDuration="7.530423621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:12.423219936 +0000 UTC m=+7.868017703" watchObservedRunningTime="2026-10-30 18:36:12.530423621 +0000 UTC m=+7.975221387"
Oct 30 18:36:12.645159 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:12.645093    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal" podStartSLOduration=7.645072011 podStartE2EDuration="7.645072011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:12.531185128 +0000 UTC m=+7.975982895" watchObservedRunningTime="2026-10-30 18:36:12.645072011 +0000 UTC m=+8.089869778"
Oct 30 18:36:13.330142 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:13.329690    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rqbk8" event={"ID":"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9","Type":"ContainerStarted","Data":"94cdc5281687177bfa2c40deaca48dd6a4c14a7f85798ca0efe9605431bd514a"}
Oct 30 18:36:13.969727 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:13.969255    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:13.969727 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:13.969338    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:13.969727 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:13.969491    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:13.969727 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:13.969551    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret podName:2601c6f1-0eb6-4038-b54c-5f8a7214db42 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:21.96953397 +0000 UTC m=+17.414331720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret") pod "global-pull-secret-syncer-55w4h" (UID: "2601c6f1-0eb6-4038-b54c-5f8a7214db42") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:13.970308 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:13.970191    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:13.970308 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:13.970255    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs podName:cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:21.970239994 +0000 UTC m=+17.415037739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs") pod "network-metrics-daemon-4zt8w" (UID: "cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:14.070390 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:14.070353    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:14.070564 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:14.070545    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:14.070659 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:14.070576    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:14.070659 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:14.070623    2571 projected.go:196] Error preparing data for projected volume kube-api-access-4hpcs for pod openshift-network-diagnostics/network-check-target-5phnp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:14.070776 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:14.070692    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs podName:92bb0644-0b77-405a-bcf2-4b88b8ac4d4d nodeName:}" failed. No retries permitted until 2026-10-30 18:36:22.070666052 +0000 UTC m=+17.515463816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hpcs" (UniqueName: "kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs") pod "network-check-target-5phnp" (UID: "92bb0644-0b77-405a-bcf2-4b88b8ac4d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:14.213857 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:14.213358    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:14.213857 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:14.213496    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:14.213857 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:14.213562    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:14.213857 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:14.213652    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:14.213857 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:14.213709    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:14.213857 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:14.213792    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:14.335291 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:14.335223    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9pbdd" event={"ID":"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600","Type":"ContainerStarted","Data":"a0c7998d62f4841569d2d70fbfbbda98f43a42054a0de3a278fa607655dcbbf6"}
Oct 30 18:36:14.338139 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:14.338109    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns" event={"ID":"12185714-c5ea-442a-9f5d-ef7af20f28c0","Type":"ContainerStarted","Data":"5bf0bb00d235934f855b7a2056710d5c01cececb52cc43dbb8a4f56a36ea7d87"}
Oct 30 18:36:14.443180 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:14.443114    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rqbk8" podStartSLOduration=4.435628958 podStartE2EDuration="9.443092251s" totalImagesPullingTime="5.007463293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:13.438646793 +0000 UTC m=+8.883444561" watchObservedRunningTime="2026-10-30 18:36:14.443092251 +0000 UTC m=+9.887890019"
Oct 30 18:36:16.057225 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.057199    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:16.059111 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.058928    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:16.168820 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.168036    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9pbdd" podStartSLOduration=5.239423984 podStartE2EDuration="11.168017283s" totalImagesPullingTime="5.928593299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:14.451828109 +0000 UTC m=+9.896625877" watchObservedRunningTime="2026-10-30 18:36:16.168017283 +0000 UTC m=+11.612815052"
Oct 30 18:36:16.196902 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.196878    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:36:16.213191 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.213148    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:16.213305 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:16.213287    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:16.213376 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.213356    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:16.213451 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:16.213425    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:16.213561 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.213482    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:16.213561 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:16.213540    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:16.348098 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.348060    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vwxwq" event={"ID":"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c","Type":"ContainerStarted","Data":"7c54cfe83294c760fb59d14f318b2ff1acb4e5fc32f86015b8f46b5884177c2a"}
Oct 30 18:36:16.351596 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.351561    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns" event={"ID":"12185714-c5ea-442a-9f5d-ef7af20f28c0","Type":"ContainerStarted","Data":"88d3560c9e2506f3c69c3c77ea66825ae8a54da1b14307bcd98650364008ca1d"}
Oct 30 18:36:16.351721 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.351631    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:16.352162 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.352145    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:16.462126 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:16.462070    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vwxwq" podStartSLOduration=2.9351748410000003 podStartE2EDuration="11.462051216s" totalImagesPullingTime="8.526876375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:16.461407471 +0000 UTC m=+11.906205242" watchObservedRunningTime="2026-10-30 18:36:16.462051216 +0000 UTC m=+11.906848984"
Oct 30 18:36:17.054093 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:17.053745    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:36:16.196897599Z","UUID":"88c6831e-043e-4690-8444-b2b19643e6f6","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:36:17.055804 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:17.055738    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:36:17.055804 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:17.055768    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:36:17.354694 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:17.354656    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sw4vz" event={"ID":"07d6f30f-5f4f-4b67-9dcc-3cae33390c84","Type":"ContainerStarted","Data":"69bec2a0082d4ecdf7e2ae1c550250d0fceb1de01023ab1a7a4dba82af2a195e"}
Oct 30 18:36:17.357353 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:17.357244    2571 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="7964e7e8d976372234681e2fea7a0b1b75d5286de7b49e7bec2029a9ab187e8b" exitCode=0
Oct 30 18:36:17.357353 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:17.357332    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84s2b" event={"ID":"65831eb9-258f-4f08-96b0-1b4556b183a7","Type":"ContainerDied","Data":"7964e7e8d976372234681e2fea7a0b1b75d5286de7b49e7bec2029a9ab187e8b"}
Oct 30 18:36:17.461142 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:17.461045    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-sw4vz" podStartSLOduration=3.46655625 podStartE2EDuration="12.461025452s" totalImagesPullingTime="8.994469202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:17.460712746 +0000 UTC m=+12.905510512" watchObservedRunningTime="2026-10-30 18:36:17.461025452 +0000 UTC m=+12.905823217"
Oct 30 18:36:18.213652 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:18.213485    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:18.213652 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:18.213506    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:18.213652 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:18.213524    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:18.213652 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:18.213618    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:18.213914 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:18.213710    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:18.213914 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:18.213771    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:18.361862 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:18.361825    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns" event={"ID":"12185714-c5ea-442a-9f5d-ef7af20f28c0","Type":"ContainerStarted","Data":"373b63bff12680bff178410e5cbf60adb4d8b8ee5f88ec970a3b4c7181b3b667"}
Oct 30 18:36:18.363911 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:18.363877    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5xvkc" event={"ID":"c50a391f-9c33-4ece-8c52-e767af7a41b1","Type":"ContainerStarted","Data":"796c1e41b447ccdf9447b4f2dacdc8372db7b59ee031b5fecfbdceb97ac65172"}
Oct 30 18:36:18.365605 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:18.365553    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rrhvc" event={"ID":"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5","Type":"ContainerStarted","Data":"01fdc61ed2f0b1e239f5e3fcedfbd652b6b050e388a0db189f1b1d804939f1e5"}
Oct 30 18:36:18.471434 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:18.471389    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns" podStartSLOduration=3.825422118 podStartE2EDuration="13.471375147s" totalImagesPullingTime="9.645953029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:18.470554942 +0000 UTC m=+13.915352709" watchObservedRunningTime="2026-10-30 18:36:18.471375147 +0000 UTC m=+13.916172912"
Oct 30 18:36:18.688272 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:18.688162    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rrhvc" podStartSLOduration=3.1168835440000002 podStartE2EDuration="13.688144787s" totalImagesPullingTime="10.571261243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:18.580522734 +0000 UTC m=+14.025320499" watchObservedRunningTime="2026-10-30 18:36:18.688144787 +0000 UTC m=+14.132942552"
Oct 30 18:36:18.688735 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:18.688708    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5xvkc" podStartSLOduration=4.896401551 podStartE2EDuration="13.688698978s" totalImagesPullingTime="8.792297427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:18.688133288 +0000 UTC m=+14.132931053" watchObservedRunningTime="2026-10-30 18:36:18.688698978 +0000 UTC m=+14.133496744"
Oct 30 18:36:19.369438 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:19.369403    2571 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="e340e1cde8f28abcbcb81359db01f063985281284687f5d9dfdb3c05ca8a6f1c" exitCode=0
Oct 30 18:36:19.370086 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:19.369511    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84s2b" event={"ID":"65831eb9-258f-4f08-96b0-1b4556b183a7","Type":"ContainerDied","Data":"e340e1cde8f28abcbcb81359db01f063985281284687f5d9dfdb3c05ca8a6f1c"}
Oct 30 18:36:20.213711 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:20.213677    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:20.213866 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:20.213751    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:20.213866 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:20.213851    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:20.213943 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:20.213869    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:20.213974 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:20.213943    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:20.214026 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:20.214009    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:21.378440 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:21.378408    2571 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="e600804fb4d72d7e653a32db58662504aac648eb226e0cea09611dddf26dd668" exitCode=0
Oct 30 18:36:21.378968 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:21.378445    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84s2b" event={"ID":"65831eb9-258f-4f08-96b0-1b4556b183a7","Type":"ContainerDied","Data":"e600804fb4d72d7e653a32db58662504aac648eb226e0cea09611dddf26dd668"}
Oct 30 18:36:22.037953 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:22.037915    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:22.038188 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:22.037982    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:22.038188 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.038077    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:22.038188 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.038082    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:22.038188 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.038138    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret podName:2601c6f1-0eb6-4038-b54c-5f8a7214db42 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:38.038121165 +0000 UTC m=+33.482918931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret") pod "global-pull-secret-syncer-55w4h" (UID: "2601c6f1-0eb6-4038-b54c-5f8a7214db42") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:22.038188 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.038154    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs podName:cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:38.038147203 +0000 UTC m=+33.482944946 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs") pod "network-metrics-daemon-4zt8w" (UID: "cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:22.138992 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:22.138958    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:22.139163 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.139145    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:22.139219 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.139173    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:22.139219 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.139187    2571 projected.go:196] Error preparing data for projected volume kube-api-access-4hpcs for pod openshift-network-diagnostics/network-check-target-5phnp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:22.139281 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.139241    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs podName:92bb0644-0b77-405a-bcf2-4b88b8ac4d4d nodeName:}" failed. No retries permitted until 2026-10-30 18:36:38.139227618 +0000 UTC m=+33.584025390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hpcs" (UniqueName: "kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs") pod "network-check-target-5phnp" (UID: "92bb0644-0b77-405a-bcf2-4b88b8ac4d4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:22.213716 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:22.213683    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:22.213906 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:22.213786    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:22.213906 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:22.213801    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:22.213906 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.213796    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:22.213906 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.213896    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:22.214095 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:22.213997    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:23.386195 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:23.386116    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj" event={"ID":"6350fe6c-a551-41dc-88a2-f15bd2360a1c","Type":"ContainerStarted","Data":"1b811dec21ddb2ac93780c597eefc8923498f31fb859249adb7ee712000e207e"}
Oct 30 18:36:23.386645 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:23.386203    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj" event={"ID":"6350fe6c-a551-41dc-88a2-f15bd2360a1c","Type":"ContainerStarted","Data":"cfbfc84261d5af04d9792cf9fce8326cb634e04a895a2baa2bb18ca41fa4e1af"}
Oct 30 18:36:23.386645 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:23.386251    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj" event={"ID":"6350fe6c-a551-41dc-88a2-f15bd2360a1c","Type":"ContainerStarted","Data":"94afd82da06d1a45e25b5d5522c0b8cae1a3b80a81cf5715bd6d5df6d8b0bd4d"}
Oct 30 18:36:23.386645 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:23.386296    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj" event={"ID":"6350fe6c-a551-41dc-88a2-f15bd2360a1c","Type":"ContainerStarted","Data":"7df1cbeb8b8148912eadfa2a54b1f598a7d6f61787f5386c5e8cd29c3ede4aa7"}
Oct 30 18:36:23.386645 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:23.386343    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj" event={"ID":"6350fe6c-a551-41dc-88a2-f15bd2360a1c","Type":"ContainerStarted","Data":"da908d4dfd7d551bf7791af94f71a1de19d0f4c19f8854a259444f55d842a9e8"}
Oct 30 18:36:23.386645 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:23.386391    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj" event={"ID":"6350fe6c-a551-41dc-88a2-f15bd2360a1c","Type":"ContainerStarted","Data":"aee253dcb57648c2632e0383d5842e5597b3dba32b3d0b7f529a215be3d48c26"}
Oct 30 18:36:24.213714 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:24.213680    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:24.213714 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:24.213702    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:24.213931 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:24.213802    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:24.213931 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:24.213841    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:24.214020 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:24.213936    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:24.214020 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:24.214005    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:24.390718 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:24.390687    2571 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="044e074736181a1d4da9eeeef9cf38a4e6a943d00f0e499edb6fe571153195b0" exitCode=0
Oct 30 18:36:24.391190 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:24.390735    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84s2b" event={"ID":"65831eb9-258f-4f08-96b0-1b4556b183a7","Type":"ContainerDied","Data":"044e074736181a1d4da9eeeef9cf38a4e6a943d00f0e499edb6fe571153195b0"}
Oct 30 18:36:25.395603 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:25.395559    2571 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="7e06b5bde3ac2010f79e949b51c0660dd42bcf8c7f1dafc30846aa9f1db32a1b" exitCode=0
Oct 30 18:36:25.396045 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:25.395643    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84s2b" event={"ID":"65831eb9-258f-4f08-96b0-1b4556b183a7","Type":"ContainerDied","Data":"7e06b5bde3ac2010f79e949b51c0660dd42bcf8c7f1dafc30846aa9f1db32a1b"}
Oct 30 18:36:25.399068 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:25.399049    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj" event={"ID":"6350fe6c-a551-41dc-88a2-f15bd2360a1c","Type":"ContainerStarted","Data":"99fc0fd7f86776201d4ef39497f88c0aae72412f65f6117e83a8d950f824123d"}
Oct 30 18:36:26.213510 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:26.213482    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:26.213680 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:26.213482    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:26.213680 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:26.213603    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:26.213680 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:26.213599    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:26.213775 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:26.213689    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:26.213775 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:26.213760    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:26.404073 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:26.404041    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84s2b" event={"ID":"65831eb9-258f-4f08-96b0-1b4556b183a7","Type":"ContainerStarted","Data":"d6b6ebb23b5069aa244f5ff6cbcc960d9ef2314d3ca2c5c70283c98b72efa7c4"}
Oct 30 18:36:26.516780 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:26.516694    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-84s2b" podStartSLOduration=9.06331882 podStartE2EDuration="21.516681408s" totalImagesPullingTime="12.453362588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:26.51665791 +0000 UTC m=+21.961455675" watchObservedRunningTime="2026-10-30 18:36:26.516681408 +0000 UTC m=+21.961479167"
Oct 30 18:36:28.213846 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:28.213673    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:28.214353 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:28.213672    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:28.214353 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:28.213921    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:28.214353 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:28.213680    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:28.214353 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:28.214036    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:28.214353 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:28.214107    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:28.411869 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:28.411836    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj" event={"ID":"6350fe6c-a551-41dc-88a2-f15bd2360a1c","Type":"ContainerStarted","Data":"29f385a2b8ffee6ec972f892ee253dc4a7e38e06218bef6a4e8a90b7804ab5cb"}
Oct 30 18:36:28.412189 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:28.412170    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:28.412273 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:28.412228    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:28.427735 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:28.427714    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:28.668597 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:28.668532    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj" podStartSLOduration=8.404657645 podStartE2EDuration="23.668515594s" totalImagesPullingTime="15.263857949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:28.621231141 +0000 UTC m=+24.066028907" watchObservedRunningTime="2026-10-30 18:36:28.668515594 +0000 UTC m=+24.113313364"
Oct 30 18:36:29.414218 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:29.414186    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:29.428537 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:29.428510    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:30.184713 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:30.184686    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-55w4h"]
Oct 30 18:36:30.184893 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:30.184791    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:30.184893 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:30.184878    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:30.187635 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:30.187610    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5phnp"]
Oct 30 18:36:30.187771 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:30.187663    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4zt8w"]
Oct 30 18:36:30.187771 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:30.187727    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:30.187858 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:30.187828    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:30.187904 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:30.187883    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:30.187953 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:30.187939    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:32.213146 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:32.213117    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:32.213525 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:32.213210    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:32.213525 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:32.213294    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:32.213525 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:32.213310    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:32.213525 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:32.213367    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:32.213525 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:32.213470    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:34.213717 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:34.213688    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:34.213717 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:34.213717    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:34.214147 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:34.213693    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:34.214147 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:34.213793    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5phnp" podUID="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d"
Oct 30 18:36:34.214147 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:34.213877    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zt8w" podUID="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f"
Oct 30 18:36:34.214147 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:34.213930    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55w4h" podUID="2601c6f1-0eb6-4038-b54c-5f8a7214db42"
Oct 30 18:36:35.825300 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:35.825136    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeReady"
Oct 30 18:36:35.825727 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:35.825391    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:36:35.967804 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:35.967764    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xngm5"]
Oct 30 18:36:35.970016 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:35.969993    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-clpsb"]
Oct 30 18:36:35.970125 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:35.970112    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:35.972092 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:35.972028    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:35.975010 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:35.974990    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2454l"]
Oct 30 18:36:35.977278 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:35.977248    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xngm5"]
Oct 30 18:36:35.977380 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:35.977328    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:35.993474 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:35.993449    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-clpsb"]
Oct 30 18:36:36.002243 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.002222    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2454l"]
Oct 30 18:36:36.019294 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.019270    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:36:36.019465 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.019448    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vtx29\""
Oct 30 18:36:36.019675 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.019663    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:36:36.020589 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.020565    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:36:36.020851 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.020840    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:36:36.020881 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.020844    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:36:36.020881 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.020861    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zmfwz\""
Oct 30 18:36:36.027155 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.027141    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-swngw\""
Oct 30 18:36:36.036122 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.036107    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:36:36.120441 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.120407    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:36:36.120620 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.120508    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:36:36.120620 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.120519    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:36:36.141150 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30e470e7-fb0d-4ce4-aa97-18d038f99fae-config-volume\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.141220 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141164    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.141220 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141194    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpg6\" (UniqueName: \"kubernetes.io/projected/63559645-8315-4941-b0ad-62405092b592-kube-api-access-srpg6\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.141316 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141261    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.141383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngsk\" (UniqueName: \"kubernetes.io/projected/30e470e7-fb0d-4ce4-aa97-18d038f99fae-kube-api-access-cngsk\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.141418 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141409    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfpzs\" (UniqueName: \"kubernetes.io/projected/aba3fd34-c628-4399-9f05-6e7711c3489f-kube-api-access-gfpzs\") pod \"ingress-canary-clpsb\" (UID: \"aba3fd34-c628-4399-9f05-6e7711c3489f\") " pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:36.141467 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aba3fd34-c628-4399-9f05-6e7711c3489f-cert\") pod \"ingress-canary-clpsb\" (UID: \"aba3fd34-c628-4399-9f05-6e7711c3489f\") " pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:36.141521 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63559645-8315-4941-b0ad-62405092b592-crio-socket\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.141573 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141522    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e470e7-fb0d-4ce4-aa97-18d038f99fae-tmp-dir\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.141573 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.141666 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.141620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63559645-8315-4941-b0ad-62405092b592-data-volume\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.213692 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.213664    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:36.213853 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.213695    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:36.213908 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.213856    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:36.242784 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.242763    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30e470e7-fb0d-4ce4-aa97-18d038f99fae-config-volume\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.242890 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.242792    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.242890 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.242817    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpg6\" (UniqueName: \"kubernetes.io/projected/63559645-8315-4941-b0ad-62405092b592-kube-api-access-srpg6\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.242890 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.242835    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.242890 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.242855    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngsk\" (UniqueName: \"kubernetes.io/projected/30e470e7-fb0d-4ce4-aa97-18d038f99fae-kube-api-access-cngsk\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.242890 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.242884    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpzs\" (UniqueName: \"kubernetes.io/projected/aba3fd34-c628-4399-9f05-6e7711c3489f-kube-api-access-gfpzs\") pod \"ingress-canary-clpsb\" (UID: \"aba3fd34-c628-4399-9f05-6e7711c3489f\") " pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.242900    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aba3fd34-c628-4399-9f05-6e7711c3489f-cert\") pod \"ingress-canary-clpsb\" (UID: \"aba3fd34-c628-4399-9f05-6e7711c3489f\") " pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.242908    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.242915    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63559645-8315-4941-b0ad-62405092b592-crio-socket\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.242948    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e470e7-fb0d-4ce4-aa97-18d038f99fae-tmp-dir\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.242984    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:36.742964802 +0000 UTC m=+32.187762565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.242993    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.243007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.243047    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls podName:30e470e7-fb0d-4ce4-aa97-18d038f99fae nodeName:}" failed. No retries permitted until 2026-10-30 18:36:36.743033489 +0000 UTC m=+32.187831237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls") pod "dns-default-xngm5" (UID: "30e470e7-fb0d-4ce4-aa97-18d038f99fae") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.243064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63559645-8315-4941-b0ad-62405092b592-data-volume\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.243071    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:36.243130 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.243113    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:36.743101326 +0000 UTC m=+32.187899093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:36.243651 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.243189    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63559645-8315-4941-b0ad-62405092b592-crio-socket\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.243651 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.243489    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63559645-8315-4941-b0ad-62405092b592-data-volume\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.243651 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.243494    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e470e7-fb0d-4ce4-aa97-18d038f99fae-tmp-dir\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.243841 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.243820    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30e470e7-fb0d-4ce4-aa97-18d038f99fae-config-volume\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.246915 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.246896    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aba3fd34-c628-4399-9f05-6e7711c3489f-cert\") pod \"ingress-canary-clpsb\" (UID: \"aba3fd34-c628-4399-9f05-6e7711c3489f\") " pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:36.263288 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.263260    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:36:36.263382 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.263331    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxkbs\""
Oct 30 18:36:36.263489 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.263474    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:36:36.263489 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.263481    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9ph7g\""
Oct 30 18:36:36.263604 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.263523    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:36:36.263604 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.263530    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5slhp\""
Oct 30 18:36:36.263701 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.263689    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:36:36.302984 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.302955    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngsk\" (UniqueName: \"kubernetes.io/projected/30e470e7-fb0d-4ce4-aa97-18d038f99fae-kube-api-access-cngsk\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.303084 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.302998    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpzs\" (UniqueName: \"kubernetes.io/projected/aba3fd34-c628-4399-9f05-6e7711c3489f-kube-api-access-gfpzs\") pod \"ingress-canary-clpsb\" (UID: \"aba3fd34-c628-4399-9f05-6e7711c3489f\") " pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:36.303084 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.303045    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpg6\" (UniqueName: \"kubernetes.io/projected/63559645-8315-4941-b0ad-62405092b592-kube-api-access-srpg6\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.590150 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.590043    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:36.613297 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.613264    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-c5q24"]
Oct 30 18:36:36.615460 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.615442    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.665091 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.665055    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 18:36:36.665091 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.665065    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 18:36:36.665232 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.665155    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 18:36:36.665232 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.665189    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-d742r\""
Oct 30 18:36:36.667937 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.667921    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 18:36:36.669465 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.669445    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-c5q24"]
Oct 30 18:36:36.747473 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.747446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxf4\" (UniqueName: \"kubernetes.io/projected/0d7bcd44-6514-4409-852d-18fb596acb10-kube-api-access-dfxf4\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.747643 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.747493    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.747643 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.747545    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d7bcd44-6514-4409-852d-18fb596acb10-signing-key\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.747643 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.747607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:36.747643 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.747637    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d7bcd44-6514-4409-852d-18fb596acb10-signing-cabundle\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.747826 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.747650    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:36.747826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.747660    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:36.747826 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.747714    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:37.74769524 +0000 UTC m=+33.192492998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:36.747826 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.747773    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:36.747826 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.747816    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:37.747804379 +0000 UTC m=+33.192602122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:36.748026 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.747865    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:36.748026 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:36.747908    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls podName:30e470e7-fb0d-4ce4-aa97-18d038f99fae nodeName:}" failed. No retries permitted until 2026-10-30 18:36:37.747895377 +0000 UTC m=+33.192693121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls") pod "dns-default-xngm5" (UID: "30e470e7-fb0d-4ce4-aa97-18d038f99fae") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:36.848826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.848744    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d7bcd44-6514-4409-852d-18fb596acb10-signing-cabundle\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.848826 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.848803    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxf4\" (UniqueName: \"kubernetes.io/projected/0d7bcd44-6514-4409-852d-18fb596acb10-kube-api-access-dfxf4\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.849216 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.848946    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d7bcd44-6514-4409-852d-18fb596acb10-signing-key\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.849413 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.849387    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d7bcd44-6514-4409-852d-18fb596acb10-signing-cabundle\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.851382 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.851361    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d7bcd44-6514-4409-852d-18fb596acb10-signing-key\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.903025 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.903000    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxf4\" (UniqueName: \"kubernetes.io/projected/0d7bcd44-6514-4409-852d-18fb596acb10-kube-api-access-dfxf4\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.925595 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.925550    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:36.963085 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:36.963013    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-clpsb"]
Oct 30 18:36:36.966824 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:36.966797    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba3fd34_c628_4399_9f05_6e7711c3489f.slice/crio-b17d1c8464ff9ee70b0e9973869054f10fd300447d425f2446ea554674ed3ca1 WatchSource:0}: Error finding container b17d1c8464ff9ee70b0e9973869054f10fd300447d425f2446ea554674ed3ca1: Status 404 returned error can't find the container with id b17d1c8464ff9ee70b0e9973869054f10fd300447d425f2446ea554674ed3ca1
Oct 30 18:36:37.287012 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:37.286982    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-c5q24"]
Oct 30 18:36:37.290549 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:37.290525    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7bcd44_6514_4409_852d_18fb596acb10.slice/crio-8e851bac2d941775b7428ed27b4b8a3328b030dbaefe5689be9b001e88e0d9b2 WatchSource:0}: Error finding container 8e851bac2d941775b7428ed27b4b8a3328b030dbaefe5689be9b001e88e0d9b2: Status 404 returned error can't find the container with id 8e851bac2d941775b7428ed27b4b8a3328b030dbaefe5689be9b001e88e0d9b2
Oct 30 18:36:37.431345 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:37.431309    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-clpsb" event={"ID":"aba3fd34-c628-4399-9f05-6e7711c3489f","Type":"ContainerStarted","Data":"b17d1c8464ff9ee70b0e9973869054f10fd300447d425f2446ea554674ed3ca1"}
Oct 30 18:36:37.432419 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:37.432397    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24" event={"ID":"0d7bcd44-6514-4409-852d-18fb596acb10","Type":"ContainerStarted","Data":"8e851bac2d941775b7428ed27b4b8a3328b030dbaefe5689be9b001e88e0d9b2"}
Oct 30 18:36:37.756234 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:37.756185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:37.756418 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:37.756259    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:37.756418 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:37.756302    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:37.756418 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:37.756316    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:37.756418 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:37.756378    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:39.756361495 +0000 UTC m=+35.201159256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:37.756551 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:37.756412    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:37.756551 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:37.756472    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:39.756456747 +0000 UTC m=+35.201254492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:37.756551 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:37.756418    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:37.756551 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:37.756501    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls podName:30e470e7-fb0d-4ce4-aa97-18d038f99fae nodeName:}" failed. No retries permitted until 2026-10-30 18:36:39.756495412 +0000 UTC m=+35.201293156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls") pod "dns-default-xngm5" (UID: "30e470e7-fb0d-4ce4-aa97-18d038f99fae") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:38.058076 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:38.057992    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:38.058914 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:38.058123    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:38.058914 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:38.058139    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:38.058914 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:38.058220    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs podName:cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f nodeName:}" failed. No retries permitted until 2026-10-30 18:37:10.058201367 +0000 UTC m=+65.502999113 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs") pod "network-metrics-daemon-4zt8w" (UID: "cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f") : secret "metrics-daemon-secret" not found
Oct 30 18:36:38.060565 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:38.060540    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:38.159388 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:38.159351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:38.162087 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:38.162066    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:38.326383 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:38.326316    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:38.336538 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:38.336511    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:38.697190 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:38.697153    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5phnp"]
Oct 30 18:36:38.738486 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:38.738458    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92bb0644_0b77_405a_bcf2_4b88b8ac4d4d.slice/crio-4197fb68ecf3305997472c27cb009428ea9709584654dc71c3fbfab5296d8f55 WatchSource:0}: Error finding container 4197fb68ecf3305997472c27cb009428ea9709584654dc71c3fbfab5296d8f55: Status 404 returned error can't find the container with id 4197fb68ecf3305997472c27cb009428ea9709584654dc71c3fbfab5296d8f55
Oct 30 18:36:38.745880 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:38.745850    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-55w4h"]
Oct 30 18:36:38.748400 ip-10-0-128-92 kubenswrapper[2571]: W1030 18:36:38.748374    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2601c6f1_0eb6_4038_b54c_5f8a7214db42.slice/crio-abca90a9c7a5c6625fd6d481e0d95f052715a43bf6d2bf6f8b729aab83bf68c8 WatchSource:0}: Error finding container abca90a9c7a5c6625fd6d481e0d95f052715a43bf6d2bf6f8b729aab83bf68c8: Status 404 returned error can't find the container with id abca90a9c7a5c6625fd6d481e0d95f052715a43bf6d2bf6f8b729aab83bf68c8
Oct 30 18:36:39.441165 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:39.441116    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-55w4h" event={"ID":"2601c6f1-0eb6-4038-b54c-5f8a7214db42","Type":"ContainerStarted","Data":"abca90a9c7a5c6625fd6d481e0d95f052715a43bf6d2bf6f8b729aab83bf68c8"}
Oct 30 18:36:39.442290 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:39.442240    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5phnp" event={"ID":"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d","Type":"ContainerStarted","Data":"4197fb68ecf3305997472c27cb009428ea9709584654dc71c3fbfab5296d8f55"}
Oct 30 18:36:39.770791 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:39.770723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:39.770886 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:39.770826    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:39.770886 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:39.770876    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:39.770995 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:39.770900    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:39.770995 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:39.770970    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls podName:30e470e7-fb0d-4ce4-aa97-18d038f99fae nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.770949421 +0000 UTC m=+39.215747166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls") pod "dns-default-xngm5" (UID: "30e470e7-fb0d-4ce4-aa97-18d038f99fae") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:39.770995 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:39.770990    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:39.771164 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:39.771040    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.771025755 +0000 UTC m=+39.215823517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:39.771164 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:39.771084    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:39.771164 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:39.771116    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.771105293 +0000 UTC m=+39.215903052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:41.452679 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:41.452634    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24" event={"ID":"0d7bcd44-6514-4409-852d-18fb596acb10","Type":"ContainerStarted","Data":"e1cec42fa59350dd38884aee0519fb2ccd7f3a2ac9a1a02af3bbfabea3f8b0c6"}
Oct 30 18:36:41.454961 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:41.454926    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-clpsb" event={"ID":"aba3fd34-c628-4399-9f05-6e7711c3489f","Type":"ContainerStarted","Data":"6b0951e641d48f615c4dedaa7bb37c7305732932e75a0c120571c7d88c7a116f"}
Oct 30 18:36:41.559459 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:41.559391    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24" podStartSLOduration=1.947917607 podStartE2EDuration="5.559372951s" totalImagesPullingTime="3.611455344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:41.558556179 +0000 UTC m=+37.003353948" watchObservedRunningTime="2026-10-30 18:36:41.559372951 +0000 UTC m=+37.004170718"
Oct 30 18:36:41.667600 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:41.666654    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-clpsb" podStartSLOduration=2.577254578 podStartE2EDuration="6.66663646s" totalImagesPullingTime="4.089381882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:41.666345949 +0000 UTC m=+37.111143715" watchObservedRunningTime="2026-10-30 18:36:41.66663646 +0000 UTC m=+37.111434228"
Oct 30 18:36:43.462378 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:43.462338    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5phnp" event={"ID":"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d","Type":"ContainerStarted","Data":"9cbb9e8dcf0b00553a421a56bcd7a9e76296a3b4c7efb85e9d18215434c3687f"}
Oct 30 18:36:43.462811 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:43.462493    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:43.569341 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:43.569277    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5phnp" podStartSLOduration=34.511523127 podStartE2EDuration="38.569261426s" totalImagesPullingTime="4.057738299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:43.568696172 +0000 UTC m=+39.013493938" watchObservedRunningTime="2026-10-30 18:36:43.569261426 +0000 UTC m=+39.014059319"
Oct 30 18:36:43.805971 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:43.805887    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:43.805971 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:43.805943    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:43.806155 ip-10-0-128-92 kubenswrapper[2571]: I1030 18:36:43.806002    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:43.806155 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:43.806063    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:43.806155 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:43.806079    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:43.806155 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:43.806104    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:43.806155 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:43.806125    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:51.806113226 +0000 UTC m=+47.250910983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:43.806155 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:43.806139    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:51.806131656 +0000 UTC m=+47.250929399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:43.806155 ip-10-0-128-92 kubenswrapper[2571]: E1030 18:36:43.806149    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls podName:30e470e7-fb0d-4ce4-aa97-18d038f99fae nodeName:}" failed. No retries permitted until 2026-10-30 18:36:51.806143531 +0000 UTC m=+47.250941276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls") pod "dns-default-xngm5" (UID: "30e470e7-fb0d-4ce4-aa97-18d038f99fae") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:44.823710 ip-10-0-128-92 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:36:44.844005 ip-10-0-128-92 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:36:44.844186 ip-10-0-128-92 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:36:44.844240 ip-10-0-128-92 systemd[1]: kubelet.service: Consumed 2.307s CPU time.
Oct 30 18:36:44.859326 ip-10-0-128-92 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:36:44.921705 ip-10-0-128-92 kubenswrapper[4391]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:44.921705 ip-10-0-128-92 kubenswrapper[4391]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:36:44.921705 ip-10-0-128-92 kubenswrapper[4391]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:44.921705 ip-10-0-128-92 kubenswrapper[4391]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:44.925044 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.924953    4391 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:44.925044 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925042    4391 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:44.925044 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925047    4391 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:44.925044 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925051    4391 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925055    4391 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925058    4391 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925061    4391 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925064    4391 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925067    4391 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925070    4391 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925073    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925076    4391 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925078    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925081    4391 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925083    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925086    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925089    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925091    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925098    4391 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925101    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925103    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925106    4391 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:44.925238 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925109    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925112    4391 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925115    4391 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925117    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925120    4391 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925123    4391 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925125    4391 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925128    4391 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925130    4391 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925133    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925136    4391 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925139    4391 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925141    4391 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925144    4391 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925148    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925152    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925154    4391 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925158    4391 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925161    4391 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925164    4391 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:44.925731 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925166    4391 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925169    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925172    4391 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925175    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925177    4391 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925180    4391 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925183    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925185    4391 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925189    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925192    4391 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925194    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925197    4391 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925199    4391 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925202    4391 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925205    4391 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925208    4391 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925210    4391 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925213    4391 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925215    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925218    4391 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:44.926234 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925221    4391 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925223    4391 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925226    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925229    4391 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925232    4391 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925234    4391 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925237    4391 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925240    4391 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925244    4391 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925248    4391 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925252    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925256    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925259    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925262    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925264    4391 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925267    4391 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925269    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925272    4391 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925275    4391 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:44.926744 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925278    4391 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925280    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925283    4391 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925286    4391 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925289    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925291    4391 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925294    4391 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925297    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925299    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925302    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925305    4391 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925307    4391 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925310    4391 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925312    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925315    4391 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925318    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925320    4391 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925323    4391 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925326    4391 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:44.927213 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925329    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925332    4391 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925334    4391 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925338    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925341    4391 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.925343    4391 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926115    4391 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926123    4391 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926127    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926131    4391 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926134    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926137    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926139    4391 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926143    4391 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926145    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926148    4391 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926151    4391 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926154    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926157    4391 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926159    4391 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:44.927734 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926162    4391 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926165    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926168    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926170    4391 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926173    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926176    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926179    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926182    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926184    4391 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926187    4391 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926189    4391 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926192    4391 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926196    4391 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926198    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926201    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926204    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926207    4391 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926209    4391 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926212    4391 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926215    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:44.928230 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926218    4391 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926220    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926223    4391 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926226    4391 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926228    4391 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926231    4391 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926233    4391 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926236    4391 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926239    4391 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926242    4391 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926245    4391 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926248    4391 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926251    4391 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926253    4391 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926256    4391 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926259    4391 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926262    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926265    4391 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926268    4391 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926270    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:44.928751 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926273    4391 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926275    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926278    4391 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926280    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926283    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926286    4391 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926288    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926291    4391 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926294    4391 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926296    4391 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926299    4391 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926301    4391 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926304    4391 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926306    4391 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926309    4391 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926311    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926314    4391 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926317    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926320    4391 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:44.929246 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926325    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926328    4391 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926330    4391 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926334    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926336    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926339    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926342    4391 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926344    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926347    4391 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926351    4391 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926355    4391 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926358    4391 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926362    4391 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926365    4391 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926368    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926371    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926374    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926377    4391 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926381    4391 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:44.929801 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926384    4391 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926386    4391 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926389    4391 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926391    4391 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926394    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926397    4391 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926399    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926402    4391 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926404    4391 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926407    4391 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926409    4391 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926412    4391 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926415    4391 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926417    4391 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926935    4391 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926942    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926946    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926948    4391 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926951    4391 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926954    4391 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:44.930270 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926957    4391 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926960    4391 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926963    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926966    4391 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926969    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926972    4391 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926974    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926977    4391 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926980    4391 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926982    4391 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926985    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926988    4391 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926991    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926993    4391 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926996    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.926999    4391 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927001    4391 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927004    4391 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927007    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:44.930811 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927009    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927012    4391 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927014    4391 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927017    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927020    4391 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927022    4391 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927025    4391 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927028    4391 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927031    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927034    4391 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927037    4391 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927040    4391 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927042    4391 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927045    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927048    4391 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927050    4391 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927059    4391 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927062    4391 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927064    4391 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927067    4391 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:44.931276 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927070    4391 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927072    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927075    4391 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927078    4391 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927080    4391 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927083    4391 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927086    4391 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927088    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927091    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927093    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927096    4391 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927099    4391 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927101    4391 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927104    4391 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927106    4391 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927111    4391 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927115    4391 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927118    4391 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927121    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927124    4391 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:44.931790 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927128    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927131    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927133    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927137    4391 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927140    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927142    4391 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927145    4391 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927147    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927151    4391 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927154    4391 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927157    4391 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927159    4391 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927162    4391 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927165    4391 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927169    4391 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927172    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927175    4391 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927179    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927182    4391 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:44.932272 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927185    4391 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927188    4391 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927190    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927193    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927195    4391 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927198    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927201    4391 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927204    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927206    4391 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927209    4391 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927211    4391 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927214    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927217    4391 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927221    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927223    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927226    4391 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927229    4391 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927232    4391 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927235    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:44.932763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927238    4391 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927241    4391 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.927244    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927332    4391 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927343    4391 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927351    4391 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927356    4391 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927360    4391 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927364    4391 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927369    4391 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927373    4391 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927376    4391 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927379    4391 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927383    4391 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927386    4391 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927390    4391 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927393    4391 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927396    4391 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927399    4391 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927402    4391 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927405    4391 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927408    4391 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927411    4391 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927414    4391 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:36:44.933330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927418    4391 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927421    4391 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927427    4391 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927430    4391 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927433    4391 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927436    4391 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927439    4391 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927443    4391 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927446    4391 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927450    4391 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927454    4391 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927459    4391 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927462    4391 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927465    4391 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927468    4391 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927471    4391 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927474    4391 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927479    4391 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927482    4391 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927485    4391 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927488    4391 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927492    4391 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927496    4391 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927499    4391 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927502    4391 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:36:44.933965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927506    4391 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927509    4391 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927512    4391 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927515    4391 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927518    4391 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927521    4391 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927523    4391 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927527    4391 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927531    4391 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927535    4391 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927539    4391 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927542    4391 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927545    4391 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927548    4391 flags.go:64] FLAG: --help="false"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927552    4391 flags.go:64] FLAG: --hostname-override="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927556    4391 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927559    4391 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927562    4391 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927565    4391 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927569    4391 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927572    4391 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927575    4391 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927598    4391 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:36:44.934597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927601    4391 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927604    4391 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927607    4391 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927610    4391 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927614    4391 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927616    4391 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927619    4391 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927622    4391 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927625    4391 flags.go:64] FLAG: --lock-file=""
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927628    4391 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927631    4391 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927635    4391 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927640    4391 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927643    4391 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927646    4391 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927649    4391 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927653    4391 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927656    4391 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927661    4391 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927664    4391 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927669    4391 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927673    4391 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927677    4391 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927680    4391 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927683    4391 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:36:44.935265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927687    4391 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927690    4391 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927693    4391 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927696    4391 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927699    4391 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927706    4391 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927709    4391 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927713    4391 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927716    4391 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927718    4391 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927722    4391 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927725    4391 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927728    4391 flags.go:64] FLAG: --port="10250"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927731    4391 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927734    4391 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-03680e6095307cc3e"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927738    4391 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927741    4391 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927744    4391 flags.go:64] FLAG: --register-node="true"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927747    4391 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927751    4391 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927754    4391 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927757    4391 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927760    4391 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927764    4391 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927767    4391 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927771    4391 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:36:44.935929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927775    4391 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927778    4391 flags.go:64] FLAG: --runonce="false"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927781    4391 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927786    4391 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927789    4391 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927792    4391 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927795    4391 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927799    4391 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927802    4391 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927805    4391 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927808    4391 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927811    4391 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927814    4391 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927817    4391 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927820    4391 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927823    4391 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927826    4391 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927832    4391 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927835    4391 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927838    4391 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927842    4391 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927844    4391 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927847    4391 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927850    4391 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927853    4391 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:36:44.936729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927856    4391 flags.go:64] FLAG: --v="2"
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927860    4391 flags.go:64] FLAG: --version="false"
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927865    4391 flags.go:64] FLAG: --vmodule=""
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927869    4391 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.927872    4391 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928018    4391 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928023    4391 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928027    4391 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928030    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928033    4391 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928036    4391 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928039    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928042    4391 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928045    4391 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928049    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928053    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928056    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928059    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928061    4391 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928064    4391 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:44.937431 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928067    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928069    4391 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928072    4391 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928074    4391 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928077    4391 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928080    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928082    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928085    4391 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928088    4391 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928091    4391 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928093    4391 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928096    4391 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928098    4391 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928101    4391 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928104    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928107    4391 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928109    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928112    4391 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928116    4391 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928118    4391 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:44.938191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928121    4391 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928124    4391 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928128    4391 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928131    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928134    4391 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928136    4391 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928139    4391 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928141    4391 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928144    4391 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928147    4391 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928150    4391 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928153    4391 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928155    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928158    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928161    4391 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928163    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928166    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928168    4391 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928171    4391 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928174    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:44.939032 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928177    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928180    4391 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928182    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928185    4391 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928188    4391 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928190    4391 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928193    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928196    4391 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928199    4391 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928204    4391 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928208    4391 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928211    4391 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928213    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928216    4391 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928221    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928224    4391 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928226    4391 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928229    4391 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928232    4391 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:44.939893 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928234    4391 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928238    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928241    4391 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928244    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928246    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928249    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928252    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928254    4391 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928257    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928259    4391 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928262    4391 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928265    4391 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928267    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928270    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928273    4391 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928275    4391 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928278    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928280    4391 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928283    4391 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:44.940529 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928286    4391 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928288    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928291    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928293    4391 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928299    4391 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928302    4391 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928305    4391 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928308    4391 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928312    4391 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928315    4391 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928317    4391 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928320    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928322    4391 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.928348    4391 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928383    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:44.941126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928387    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928392    4391 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928395    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928398    4391 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928402    4391 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928405    4391 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928408    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928411    4391 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928414    4391 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928419    4391 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928422    4391 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928424    4391 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928427    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928430    4391 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928433    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928435    4391 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928438    4391 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928441    4391 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928443    4391 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:44.941613 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928446    4391 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928449    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928451    4391 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928456    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928459    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928461    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928466    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928468    4391 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928471    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928474    4391 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928477    4391 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928480    4391 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928482    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928485    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928487    4391 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928491    4391 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928493    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928496    4391 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928499    4391 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:44.942096 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928501    4391 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928504    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928507    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928509    4391 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928513    4391 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928517    4391 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928520    4391 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928523    4391 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928526    4391 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928529    4391 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928531    4391 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928534    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928537    4391 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928540    4391 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928542    4391 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928545    4391 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928550    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928553    4391 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928555    4391 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:44.942573 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928559    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928562    4391 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928565    4391 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928568    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928571    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928574    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928591    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928596    4391 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928600    4391 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928605    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928609    4391 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928611    4391 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928614    4391 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928617    4391 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928619    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928622    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928625    4391 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928628    4391 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928630    4391 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928633    4391 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:44.943049 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928636    4391 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928638    4391 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928641    4391 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928643    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928646    4391 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928649    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928651    4391 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928654    4391 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928656    4391 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928660    4391 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928663    4391 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928666    4391 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928670    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928673    4391 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928675    4391 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928678    4391 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928681    4391 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928683    4391 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928686    4391 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928688    4391 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:44.943528 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928691    4391 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928694    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928696    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928699    4391 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928702    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928705    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928707    4391 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.928716    4391 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.932346    4391 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.932360    4391 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.932376    4391 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.932381    4391 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932434    4391 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932440    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932444    4391 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932448    4391 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932451    4391 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932453    4391 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932456    4391 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:44.944031 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932459    4391 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932462    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932465    4391 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932468    4391 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932471    4391 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932474    4391 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932476    4391 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932479    4391 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932483    4391 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932487    4391 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932490    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932492    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932495    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932498    4391 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932501    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932504    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932508    4391 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932511    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932514    4391 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:44.944598 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932517    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932520    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932523    4391 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932525    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932528    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932531    4391 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932534    4391 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932536    4391 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932539    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932542    4391 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932544    4391 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932547    4391 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932550    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932552    4391 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932555    4391 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932557    4391 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932560    4391 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932562    4391 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932565    4391 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932567    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:44.945125 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932570    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932574    4391 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932597    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932601    4391 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932603    4391 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932606    4391 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932609    4391 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932612    4391 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932615    4391 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932617    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932620    4391 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932623    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932626    4391 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932628    4391 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932631    4391 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932634    4391 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932636    4391 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932639    4391 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932642    4391 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932644    4391 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:44.945689 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932647    4391 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932649    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932652    4391 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932654    4391 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932657    4391 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932660    4391 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932662    4391 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932665    4391 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932667    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932670    4391 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932672    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932675    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932678    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932681    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932683    4391 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932688    4391 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932691    4391 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932695    4391 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932698    4391 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:44.946181 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932700    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932703    4391 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932706    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932708    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932712    4391 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932714    4391 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932717    4391 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932720    4391 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932722    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932725    4391 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932727    4391 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932730    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932732    4391 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932735    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932737    4391 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932740    4391 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932742    4391 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932745    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932748    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:44.946652 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932750    4391 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932753    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.932781    4391 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932915    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932920    4391 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932923    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932926    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932929    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932932    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932935    4391 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932938    4391 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932940    4391 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932943    4391 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932946    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932948    4391 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932951    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:44.947146 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932953    4391 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932956    4391 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932959    4391 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932962    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932964    4391 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932968    4391 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932972    4391 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932974    4391 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932977    4391 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932980    4391 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932983    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932985    4391 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932988    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932990    4391 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932993    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932996    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.932998    4391 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933001    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933003    4391 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:44.947532 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933006    4391 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933008    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933011    4391 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933014    4391 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933016    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933019    4391 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933022    4391 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933024    4391 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933027    4391 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933029    4391 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933032    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933034    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933037    4391 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933040    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933042    4391 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933045    4391 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933048    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933050    4391 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933053    4391 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:44.948015 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933056    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933058    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933061    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933063    4391 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933066    4391 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933068    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933071    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933073    4391 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933077    4391 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933079    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933082    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933085    4391 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933087    4391 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933091    4391 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933095    4391 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933098    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933101    4391 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933104    4391 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933106    4391 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:44.948476 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933109    4391 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933111    4391 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933114    4391 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933116    4391 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933119    4391 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933122    4391 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933124    4391 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933127    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933129    4391 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933132    4391 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933134    4391 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933138    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933140    4391 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933143    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933145    4391 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933148    4391 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933151    4391 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933153    4391 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933156    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933159    4391 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:44.948980 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933161    4391 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933164    4391 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933166    4391 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933172    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933175    4391 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933177    4391 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933180    4391 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933183    4391 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933186    4391 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933188    4391 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933191    4391 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933193    4391 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933196    4391 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933198    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933201    4391 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933204    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:44.949479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.933225    4391 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933257    4391 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933261    4391 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933265    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933267    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933270    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933273    4391 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933276    4391 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933278    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933281    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933284    4391 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933287    4391 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933290    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933292    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933295    4391 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933297    4391 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933301    4391 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933305    4391 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933309    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933313    4391 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:44.950016 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933317    4391 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933319    4391 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933322    4391 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933325    4391 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933327    4391 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933330    4391 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933333    4391 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933336    4391 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933339    4391 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933342    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933345    4391 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933347    4391 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933350    4391 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933353    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933356    4391 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933359    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933361    4391 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933364    4391 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933366    4391 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933369    4391 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:44.950543 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933371    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933375    4391 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933377    4391 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933380    4391 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933382    4391 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933387    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933390    4391 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933392    4391 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933395    4391 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933399    4391 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933402    4391 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933405    4391 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933408    4391 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933411    4391 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933413    4391 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933416    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933418    4391 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933421    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933424    4391 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933426    4391 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:44.951142 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933429    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933432    4391 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933434    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933437    4391 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933439    4391 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933442    4391 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933445    4391 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933447    4391 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933450    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933452    4391 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933455    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933457    4391 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933460    4391 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933463    4391 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933466    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933468    4391 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933472    4391 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933477    4391 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933480    4391 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:44.951657 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933483    4391 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933486    4391 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933489    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933491    4391 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933495    4391 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933498    4391 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933501    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933503    4391 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933506    4391 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933508    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933511    4391 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933514    4391 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933516    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933519    4391 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933521    4391 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933524    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933528    4391 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933531    4391 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933533    4391 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:44.952132 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933536    4391 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933538    4391 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933541    4391 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933544    4391 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933546    4391 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933549    4391 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933551    4391 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933554    4391 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:44.933556    4391 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.933620    4391 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.934554    4391 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.934627    4391 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.934795    4391 server.go:1014] "Starting client certificate rotation"
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.934951    4391 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.935009    4391 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:31:05 +0000 UTC" deadline="2028-07-12 22:41:00.442660182 +0000 UTC"
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.935044    4391 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14908h4m15.507619857s"
Oct 30 18:36:44.952614 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.935318    4391 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:44.952997 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.936361    4391 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:44.952997 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.938262    4391 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:36:44.952997 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.940354    4391 log.go:25] "Validated CRI v1 image API"
Oct 30 18:36:44.952997 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.941095    4391 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:36:44.952997 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.942552    4391 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8f5b8466-d342-4f5d-b3a4-1c2358c3bf9b:/dev/nvme0n1p3 97db2337-79f5-4adf-8f0c-2fabe5b9d224:/dev/nvme0n1p4]
Oct 30 18:36:44.953144 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.942573    4391 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1115cd671725385db8c19e02126d4a56396798424abc30ad0bbc5c2eae3c2e63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1115cd671725385db8c19e02126d4a56396798424abc30ad0bbc5c2eae3c2e63/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bb931b9e061af7cd61facf138ea815b12ee4acec95ca94560c27f9666c09476/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bb931b9e061af7cd61facf138ea815b12ee4acec95ca94560c27f9666c09476/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/359d26e716db916f5470c236c86b93b32950f8aa1b8d36310d295bd2fcfc50f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/359d26e716db916f5470c236c86b93b32950f8aa1b8d36310d295bd2fcfc50f2/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3eeb1c8d921c3819becb1e00c18a85617da55d8965045cfbce0e368a84e1bd8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3eeb1c8d921c3819becb1e00c18a85617da55d8965045cfbce0e368a84e1bd8f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4197fb68ecf3305997472c27cb009428ea9709584654dc71c3fbfab5296d8f55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4197fb68ecf3305997472c27cb009428ea9709584654dc71c3fbfab5296d8f55/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4406b0a0d4aaf7f96fcde8da9366d4fd03593a4643176dc9800015a852daabe5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4406b0a0d4aaf7f96fcde8da9366d4fd03593a4643176dc9800015a852daabe5/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68cadacbe7916641daa1f4eec003e4ed0cf78fc56c3a12501e8d13700c88dba8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68cadacbe7916641daa1f4eec003e4ed0cf78fc56c3a12501e8d13700c88dba8/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7671310b8bd536d1490c6752b1cfbd4d4a092141a5d767a83f57e6c0a7b0d4ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7671310b8bd536d1490c6752b1cfbd4d4a092141a5d767a83f57e6c0a7b0d4ee/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b70d86fbf99aa801145917ef02735a548b46e2c2eb41d357f0722007c48e301/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b70d86fbf99aa801145917ef02735a548b46e2c2eb41d357f0722007c48e301/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/831b762ae9cc950299f132beb1715e116a8afc93faa442c3c7f8ea4eed440d64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/831b762ae9cc950299f132beb1715e116a8afc93faa442c3c7f8ea4eed440d64/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e851bac2d941775b7428ed27b4b8a3328b030dbaefe5689be9b001e88e0d9b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e851bac2d941775b7428ed27b4b8a3328b030dbaefe5689be9b001e88e0d9b2/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abca90a9c7a5c6625fd6d481e0d95f052715a43bf6d2bf6f8b729aab83bf68c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abca90a9c7a5c6625fd6d481e0d95f052715a43bf6d2bf6f8b729aab83bf68c8/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abe7d5a8d94625e7d5425c441f0d6f029510612afca0199219d74290fac515ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abe7d5a8d94625e7d5425c441f0d6f029510612afca0199219d74290fac515ec/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b17d1c8464ff9ee70b0e9973869054f10fd300447d425f2446ea554674ed3ca1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b17d1c8464ff9ee70b0e9973869054f10fd300447d425f2446ea554674ed3ca1/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07d6f30f-5f4f-4b67-9dcc-3cae33390c84/volumes/kubernetes.io~projected/kube-api-access-7jhk6:{mountpoint:/var/lib/kubelet/pods/07d6f30f-5f4f-4b67-9dcc-3cae33390c84/volumes/kubernetes.io~projected/kube-api-access-7jhk6 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d7bcd44-6514-4409-852d-18fb596acb10/volumes/kubernetes.io~projected/kube-api-access-dfxf4:{mountpoint:/var/lib/kubelet/pods/0d7bcd44-6514-4409-852d-18fb596acb10/volumes/kubernetes.io~projected/kube-api-access-dfxf4 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d7bcd44-6514-4409-852d-18fb596acb10/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0d7bcd44-6514-4409-852d-18fb596acb10/volumes/kubernetes.io~secret/signing-key major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12185714-c5ea-442a-9f5d-ef7af20f28c0/volumes/kubernetes.io~projected/kube-api-access-t92v5:{mountpoint:/var/lib/kubelet/pods/12185714-c5ea-442a-9f5d-ef7af20f28c0/volumes/kubernetes.io~projected/kube-api-access-t92v5 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2601c6f1-0eb6-4038-b54c-5f8a7214db42/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2601c6f1-0eb6-4038-b54c-5f8a7214db42/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30e470e7-fb0d-4ce4-aa97-18d038f99fae/volumes/kubernetes.io~projected/kube-api-access-cngsk:{mountpoint:/var/lib/kubelet/pods/30e470e7-fb0d-4ce4-aa97-18d038f99fae/volumes/kubernetes.io~projected/kube-api-access-cngsk major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600/volumes/kubernetes.io~projected/kube-api-access-z6qxp:{mountpoint:/var/lib/kubelet/pods/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600/volumes/kubernetes.io~projected/kube-api-access-z6qxp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6350fe6c-a551-41dc-88a2-f15bd2360a1c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6350fe6c-a551-41dc-88a2-f15bd2360a1c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6350fe6c-a551-41dc-88a2-f15bd2360a1c/volumes/kubernetes.io~projected/kube-api-access-4r7z9:{mountpoint:/var/lib/kubelet/pods/6350fe6c-a551-41dc-88a2-f15bd2360a1c/volumes/kubernetes.io~projected/kube-api-access-4r7z9 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6350fe6c-a551-41dc-88a2-f15bd2360a1c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6350fe6c-a551-41dc-88a2-f15bd2360a1c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63559645-8315-4941-b0ad-62405092b592/volumes/kubernetes.io~projected/kube-api-access-srpg6:{mountpoint:/var/lib/kubelet/pods/63559645-8315-4941-b0ad-62405092b592/volumes/kubernetes.io~projected/kube-api-access-srpg6 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65831eb9-258f-4f08-96b0-1b4556b183a7/volumes/kubernetes.io~projected/kube-api-access-mltx7:{mountpoint:/var/lib/kubelet/pods/65831eb9-258f-4f08-96b0-1b4556b183a7/volumes/kubernetes.io~projected/kube-api-access-mltx7 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d/volumes/kubernetes.io~projected/kube-api-access-4hpcs:{mountpoint:/var/lib/kubelet/pods/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d/volumes/kubernetes.io~projected/kube-api-access-4hpcs major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aba3fd34-c628-4399-9f05-6e7711c3489f/volumes/kubernetes.io~projected/kube-api-access-gfpzs:{mountpoint:/var/lib/kubelet/pods/aba3fd34-c628-4399-9f05-6e7711c3489f/volumes/kubernetes.io~projected/kube-api-access-gfpzs major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aba3fd34-c628-4399-9f05-6e7711c3489f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/aba3fd34-c628-4399-9f05-6e7711c3489f/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c/volumes/kubernetes.io~projected/kube-api-access-f2nlc:{mountpoint:/var/lib/kubelet/pods/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c/volumes/kubernetes.io~projected/kube-api-access-f2nlc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c50a391f-9c33-4ece-8c52-e767af7a41b1/volumes/kubernetes.io~projected/kube-api-access-xp648:{mountpoint:/var/lib/kubelet/pods/c50a391f-9c33-4ece-8c52-e767af7a41b1/volumes/kubernetes.io~projected/kube-api-access-xp648 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f/volumes/kubernetes.io~projected/kube-api-access-vp9pl:{mountpoint:/var/lib/kubelet/pods/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f/volumes/kubernetes.io~projected/kube-api-access-vp9pl major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5/volumes/kubernetes.io~projected/kube-api-access-n6bj4:{mountpoint:/var/lib/kubelet/pods/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5/volumes/kubernetes.io~projected/kube-api-access-n6bj4 major:0 minor:53 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e0d9d74d232432c1527a8305097dd5dda118137e040ec77af0e69181a44ac1a5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1540d477952f6e34895bc309d6ee3e128161220a4bfbd9fcd773dc582221706c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8be7b9518f5e5bcaef2124f0f993c539980fa84bd14a22c98d573f78de8ccb2c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/1b339b7e598235dd322f764d2023f9ba3667d7284525eae437b66085232a7c66/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1caebb0f4393a07f27ea5de35c219574009d664da7d3306897e3417f572ebea8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b7d2757e0fbef5c91a7b7de01ec16f5a1eb557d5001804883508995ab31baa70/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ea5aacdbd27d654a50610ce602524dc0ce8bb49d53c6d0dd557c25da97924584/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/27b159bc7471adfd80ccc239fe845cce8d22286be707e1c10962ab22a2f14e9b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/79d6e3ef4a61385545081cf709e13b0589f0cf6e1719eaaf07148c2eb43feaf2/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/da340741d13cdc5a2829a10a40dcbc1b9a5c74f50b2714f128695791f2ae94ee/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a72180354e59134b61b9ebcab23c38e466ca562243f2d30a453124d72ae102fa/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/1a471ee895fb762251cb28ee02fa3ce67dfacc79325d09c39867559427c6177d/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d35df15637ff0dc94f5b4470cd230f7d4c7e9916a7543cd8eea87415e48918e1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9c2e3826b71cacab3fa08755453b792ddafcc77d768c4b03be9085011fd49d8a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e0d0034df8cf95cea36825677f7a233ebaeafabcdb62d8c0da0d0144c2e6a1ea/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/10d3bd9d54803eabcb12eac0b7c5050d686d352d3d0d6be735c77ad570f50f1b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f9908cd113fb72d7b1d78666f2ada59f54391b99686e51135a606379b45252de/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/32732d7dfc78f8f44eb95c4cb284c14149d5b7b5248e50571d8fdc76e93167db/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/561be12079966e64d36db3ce5a203841cef2231efb18c81a76c8cb9367541a0a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/8c3d0748372271d601a43b56c48c50bd2a32d2d55d9e051eb577a2988b3b2b6a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e455c4889624e53467811886632c2d37fb2696ca81b71905c1a61222cdad1ee7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/02c22c9d752646c0c15c86dcbb70c17b1c59204b3646ba6d4b99934957568ec5/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/09dc0cd1af5a135cc81127fc7d16dbd6cde625adc95ab538bf7e26b1af21a1b9/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/29f249880baba818557ce2d9668ea0083d47c8d83d90f6cba7328a3eba7e9ae5/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/93b6169d44e60a5b5005eb410adc13d2acf3d1b3630e4f585f0b476a381c3c2e/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/3eb62d93c16a7ce36e7d6d2d2a63c1c24dda82fc6f319b1d96570f7f342de77a/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/cd2c1a2191d7d03a7aa98b90331f778c4b7cb64f6d96aeb615767aec255b76d6/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/5927978847f3802e1a21f7c9033c5cde414e568b7412edc45307b546baf9aa9d/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/8c8d29d707e7eabb89dd7dfd4a8359e1d554494514323e788edf1f4c30e444ca/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/d14fc230bffcc4f25136fa91e9a53f269c412d7af7ee2dc01b21c0aa012a697f/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/520d0d7c41f23949fb05b23f1e81616671cd96bef575e2bbbc1e3dc72248ac22/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/010c6e2783459d22c81461624a515eecb2782f59691d47560ffa1621bc038b2e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/85afee73a15f6f3ae75d6671824332adf2a6dd48ff0e561fc47987fb5311be06/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c6d14f9585cc8f5e21c11e0a0679c5a52dfc9c022b9fa044407bb636fc7b44bf/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/0059d07e53707f716b98035bb1656ab543f17ba582ce11068a4ed8d4769573ce/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/741869da6d634cc2fc53f59e288f9b3f494b624b98d836585a82f2beeaf1cb62/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/497f38c6d4cea90826a23594d18d38edadbe88c04558e0743a7bd056db9e5929/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/7113897d2c7eb41518dc41df1bd2f1702cbe379b3570196405142564656b27e9/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6cef7a1ef05554b18ddfcdba47ee3fa503da31dd4e58f172f7b897dbebb537ab/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.948469    4391 manager.go:217] Machine: {Timestamp:2026-10-30 18:36:44.948017586 +0000 UTC m=+0.058750860 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099608 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2610c0e0988a0f9bcca89acb9dc73c SystemUUID:ec2610c0-e098-8a0f-9bcc-a89acb9dc73c BootID:c87c4750-4d78-4fc5-8953-afd9eaafc18c Filesystems:[{Device:/var/lib/kubelet/pods/0d7bcd44-6514-4409-852d-18fb596acb10/volumes/kubernetes.io~projected/kube-api-access-dfxf4 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b70d86fbf99aa801145917ef02735a548b46e2c2eb41d357f0722007c48e301/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68cadacbe7916641daa1f4eec003e4ed0cf78fc56c3a12501e8d13700c88dba8/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2601c6f1-0eb6-4038-b54c-5f8a7214db42/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6350fe6c-a551-41dc-88a2-f15bd2360a1c/volumes/kubernetes.io~projected/kube-api-access-4r7z9 DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aba3fd34-c628-4399-9f05-6e7711c3489f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aba3fd34-c628-4399-9f05-6e7711c3489f/volumes/kubernetes.io~projected/kube-api-access-gfpzs DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d7bcd44-6514-4409-852d-18fb596acb10/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f/volumes/kubernetes.io~projected/kube-api-access-vp9pl DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abe7d5a8d94625e7d5425c441f0d6f029510612afca0199219d74290fac515ec/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/359d26e716db916f5470c236c86b93b32950f8aa1b8d36310d295bd2fcfc50f2/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c50a391f-9c33-4ece-8c52-e767af7a41b1/volumes/kubernetes.io~projected/kube-api-access-xp648 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/831b762ae9cc950299f132beb1715e116a8afc93faa442c3c7f8ea4eed440d64/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4406b0a0d4aaf7f96fcde8da9366d4fd03593a4643176dc9800015a852daabe5/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e851bac2d941775b7428ed27b4b8a3328b030dbaefe5689be9b001e88e0d9b2/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1115cd671725385db8c19e02126d4a56396798424abc30ad0bbc5c2eae3c2e63/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abca90a9c7a5c6625fd6d481e0d95f052715a43bf6d2bf6f8b729aab83bf68c8/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4197fb68ecf3305997472c27cb009428ea9709584654dc71c3fbfab5296d8f55/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6350fe6c-a551-41dc-88a2-f15bd2360a1c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3eeb1c8d921c3819becb1e00c18a85617da55d8965045cfbce0e368a84e1bd8f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b17d1c8464ff9ee70b0e9973869054f10fd300447d425f2446ea554674ed3ca1/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/07d6f30f-5f4f-4b67-9dcc-3cae33390c84/volumes/kubernetes.io~projected/kube-api-access-7jhk6 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30e470e7-fb0d-4ce4-aa97-18d038f99fae/volumes/kubernetes.io~projected/kube-api-access-cngsk DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65831eb9-258f-4f08-96b0-1b4556b183a7/volumes/kubernetes.io~projected/kube-api-access-mltx7 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63559645-8315-4941-b0ad-62405092b592/volumes/kubernetes.io~projected/kube-api-access-srpg6 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5/volumes/kubernetes.io~projected/kube-api-access-n6bj4 DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600/volumes/kubernetes.io~projected/kube-api-access-z6qxp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bb931b9e061af7cd61facf138ea815b12ee4acec95ca94560c27f9666c09476/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6350fe6c-a551-41dc-88a2-f15bd2360a1c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d/volumes/kubernetes.io~projected/kube-api-access-4hpcs DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12185714-c5ea-442a-9f5d-ef7af20f28c0/volumes/kubernetes.io~projected/kube-api-access-t92v5 DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c/volumes/kubernetes.io~projected/kube-api-access-f2nlc DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7671310b8bd536d1490c6752b1cfbd4d4a092141a5d767a83f57e6c0a7b0d4ee/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:4197fb68ecf3305 MacAddress:3e:08:77:1a:b4:13 Speed:10000 Mtu:8901} {Name:8e851bac2d94177 MacAddress:66:86:7f:20:bf:be Speed:10000 Mtu:8901} {Name:abca90a9c7a5c66 MacAddress:82:97:a7:40:be:d9 Speed:10000 Mtu:8901} {Name:b17d1c8464ff9ee MacAddress:f2:27:16:30:f9:ea Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d3:e4:1b:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:b6:a7:74:00:36:b3 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:d3:e4:1b:7d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:6e:5a:2b:65:c4:7b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:02:81:3c:0d:03:8a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.948716    4391 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.948758    4391 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949018    4391 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949037    4391 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-92.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949169    4391 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949177    4391 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949191    4391 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949210    4391 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949287    4391 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949436    4391 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949541    4391 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949573    4391 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949599    4391 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949609    4391 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949618    4391 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.949625    4391 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.950219    4391 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.950232    4391 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.950728    4391 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.952825    4391 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.952834    4391 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.952956    4391 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953176    4391 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:36:44.953287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953208    4391 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953306    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953318    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953326    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953331    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953338    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953346    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953355    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953363    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953370    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953376    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953384    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953394    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953426    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:36:44.954068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.953433    4391 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:36:44.956839 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.956827    4391 server.go:1263] "Started kubelet"
Oct 30 18:36:44.956959 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.956929    4391 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:36:44.957406 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.957047    4391 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:36:44.957406 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.957124    4391 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:36:44.957559 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.957503    4391 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:36:44.957439 ip-10-0-128-92 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:36:44.961345 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.960260    4391 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:36:44.963658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.963401    4391 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:36:44.963658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.963417    4391 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:44.963658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.963475    4391 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:31:06 +0000 UTC" deadline="2028-08-09 08:44:25.370073788 +0000 UTC"
Oct 30 18:36:44.963658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.963494    4391 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15566h7m40.40658327s"
Oct 30 18:36:44.963658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.963505    4391 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:36:44.963658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.963519    4391 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:36:44.963658 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:44.963526    4391 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-92.us-west-2.compute.internal\" not found"
Oct 30 18:36:44.964062 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.964045    4391 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:36:44.965573 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.965554    4391 factory.go:55] Registering systemd factory
Oct 30 18:36:44.965674 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.965595    4391 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:36:44.965949 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.965933    4391 factory.go:153] Registering CRI-O factory
Oct 30 18:36:44.965949 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.965951    4391 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:36:44.966098 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.966002    4391 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:36:44.966098 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.966020    4391 factory.go:103] Registering Raw factory
Oct 30 18:36:44.966098 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.966033    4391 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:36:44.966292 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.966280    4391 manager.go:319] Starting recovery of all containers
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967157    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d7bcd44-6514-4409-852d-18fb596acb10" volumeName="kubernetes.io/secret/0d7bcd44-6514-4409-852d-18fb596acb10-signing-key" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967197    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2601c6f1-0eb6-4038-b54c-5f8a7214db42" volumeName="kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967212    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6350fe6c-a551-41dc-88a2-f15bd2360a1c" volumeName="kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967238    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9" volumeName="kubernetes.io/configmap/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967251    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b976d8d0-4b1e-43a8-874d-94d3ecde6a3c" volumeName="kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967263    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e470e7-fb0d-4ce4-aa97-18d038f99fae" volumeName="kubernetes.io/configmap/30e470e7-fb0d-4ce4-aa97-18d038f99fae-config-volume" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967288    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c50a391f-9c33-4ece-8c52-e767af7a41b1" volumeName="kubernetes.io/projected/c50a391f-9c33-4ece-8c52-e767af7a41b1-kube-api-access-xp648" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967321    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5" volumeName="kubernetes.io/projected/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-kube-api-access-n6bj4" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967337    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6350fe6c-a551-41dc-88a2-f15bd2360a1c" volumeName="kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-env-overrides" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967361    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6350fe6c-a551-41dc-88a2-f15bd2360a1c" volumeName="kubernetes.io/projected/6350fe6c-a551-41dc-88a2-f15bd2360a1c-kube-api-access-4r7z9" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967376    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63559645-8315-4941-b0ad-62405092b592" volumeName="kubernetes.io/empty-dir/63559645-8315-4941-b0ad-62405092b592-data-volume" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967391    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65831eb9-258f-4f08-96b0-1b4556b183a7" volumeName="kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967427    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65831eb9-258f-4f08-96b0-1b4556b183a7" volumeName="kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967440    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65831eb9-258f-4f08-96b0-1b4556b183a7" volumeName="kubernetes.io/projected/65831eb9-258f-4f08-96b0-1b4556b183a7-kube-api-access-mltx7" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967498    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92bb0644-0b77-405a-bcf2-4b88b8ac4d4d" volumeName="kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967520    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aba3fd34-c628-4399-9f05-6e7711c3489f" volumeName="kubernetes.io/secret/aba3fd34-c628-4399-9f05-6e7711c3489f-cert" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967530    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c50a391f-9c33-4ece-8c52-e767af7a41b1" volumeName="kubernetes.io/configmap/c50a391f-9c33-4ece-8c52-e767af7a41b1-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967539    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f" volumeName="kubernetes.io/projected/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-kube-api-access-vp9pl" seLinuxMountContext=""
Oct 30 18:36:44.967549 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967559    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5" volumeName="kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-tuned" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967570    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07d6f30f-5f4f-4b67-9dcc-3cae33390c84" volumeName="kubernetes.io/empty-dir/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-tmp-dir" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967594    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12185714-c5ea-442a-9f5d-ef7af20f28c0" volumeName="kubernetes.io/projected/12185714-c5ea-442a-9f5d-ef7af20f28c0-kube-api-access-t92v5" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967618    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e470e7-fb0d-4ce4-aa97-18d038f99fae" volumeName="kubernetes.io/projected/30e470e7-fb0d-4ce4-aa97-18d038f99fae-kube-api-access-cngsk" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967636    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6350fe6c-a551-41dc-88a2-f15bd2360a1c" volumeName="kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-config" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967646    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63559645-8315-4941-b0ad-62405092b592" volumeName="kubernetes.io/projected/63559645-8315-4941-b0ad-62405092b592-kube-api-access-srpg6" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967666    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b976d8d0-4b1e-43a8-874d-94d3ecde6a3c" volumeName="kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967696    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5" volumeName="kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-tmp" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967706    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07d6f30f-5f4f-4b67-9dcc-3cae33390c84" volumeName="kubernetes.io/projected/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-kube-api-access-7jhk6" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967729    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d7bcd44-6514-4409-852d-18fb596acb10" volumeName="kubernetes.io/configmap/0d7bcd44-6514-4409-852d-18fb596acb10-signing-cabundle" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967788    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57a4e656-ef45-4ef7-b2f5-bbb46c7fc600" volumeName="kubernetes.io/projected/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-kube-api-access-z6qxp" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967801    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9" volumeName="kubernetes.io/secret/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-agent-certs" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967839    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aba3fd34-c628-4399-9f05-6e7711c3489f" volumeName="kubernetes.io/projected/aba3fd34-c628-4399-9f05-6e7711c3489f-kube-api-access-gfpzs" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967851    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e470e7-fb0d-4ce4-aa97-18d038f99fae" volumeName="kubernetes.io/empty-dir/30e470e7-fb0d-4ce4-aa97-18d038f99fae-tmp-dir" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967873    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6350fe6c-a551-41dc-88a2-f15bd2360a1c" volumeName="kubernetes.io/secret/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967914    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65831eb9-258f-4f08-96b0-1b4556b183a7" volumeName="kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967925    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b976d8d0-4b1e-43a8-874d-94d3ecde6a3c" volumeName="kubernetes.io/projected/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-kube-api-access-f2nlc" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967936    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d7bcd44-6514-4409-852d-18fb596acb10" volumeName="kubernetes.io/projected/0d7bcd44-6514-4409-852d-18fb596acb10-kube-api-access-dfxf4" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967957    4391 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57a4e656-ef45-4ef7-b2f5-bbb46c7fc600" volumeName="kubernetes.io/configmap/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-serviceca" seLinuxMountContext=""
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967969    4391 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:36:44.968404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.967976    4391 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:36:44.997010 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.996960    4391 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:36:44.998217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.998193    4391 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:36:44.998217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.998219    4391 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:36:44.998343 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:44.998238    4391 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:36:44.998343 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:44.998274    4391 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:36:45.012036 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.012003    4391 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="7e06b5bde3ac2010f79e949b51c0660dd42bcf8c7f1dafc30846aa9f1db32a1b" exitCode=0
Oct 30 18:36:45.012036 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.012036    4391 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="044e074736181a1d4da9eeeef9cf38a4e6a943d00f0e499edb6fe571153195b0" exitCode=0
Oct 30 18:36:45.012197 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.012047    4391 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="e600804fb4d72d7e653a32db58662504aac648eb226e0cea09611dddf26dd668" exitCode=0
Oct 30 18:36:45.012197 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.012056    4391 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="e340e1cde8f28abcbcb81359db01f063985281284687f5d9dfdb3c05ca8a6f1c" exitCode=0
Oct 30 18:36:45.012197 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.012068    4391 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="7964e7e8d976372234681e2fea7a0b1b75d5286de7b49e7bec2029a9ab187e8b" exitCode=0
Oct 30 18:36:45.012197 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.012076    4391 generic.go:356] "Generic (PLEG): container finished" podID="65831eb9-258f-4f08-96b0-1b4556b183a7" containerID="fdd9fe55abff8e5ff91754dd02a89906836215bb4aebc866f88c712768c90086" exitCode=0
Oct 30 18:36:45.031823 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.031802    4391 manager.go:324] Recovery completed
Oct 30 18:36:45.036106 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.036079    4391 generic.go:356] "Generic (PLEG): container finished" podID="e4b49969aba00a9d5dd95ce58685c160" containerID="5a3e22e86f441d0d9181473e64dec9ecfcc223f78239b07658881d878a07bc4a" exitCode=0
Oct 30 18:36:45.045126 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.045111    4391 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:45.046116 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.046101    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:45.046175 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.046129    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:45.046175 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.046139    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:45.047371 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047354    4391 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:36:45.047371 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047368    4391 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:36:45.047495 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047383    4391 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:45.047495 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047387    4391 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:36:45.047670 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047652    4391 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:36:45.047731 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047672    4391 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:36:45.047838 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047826    4391 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:36:45.047887 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047839    4391 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:36:45.047887 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047848    4391 policy_none.go:50] "Start"
Oct 30 18:36:45.047887 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047861    4391 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:36:45.047887 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.047876    4391 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:36:45.048118 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.048099    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:45.048217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.048131    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:45.048217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.048145    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:45.048922 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.048903    4391 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:36:45.048987 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.048925    4391 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:36:45.048987 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.048934    4391 policy_none.go:44] "Start"
Oct 30 18:36:45.052899 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.052875    4391 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:36:45.057041 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.057027    4391 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:36:45.057121 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:45.057058    4391 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:36:45.057121 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.057069    4391 server.go:87] "Starting device plugin registration server"
Oct 30 18:36:45.057381 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.057369    4391 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:36:45.057424 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.057384    4391 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:36:45.057503 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.057484    4391 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:36:45.057656 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.057620    4391 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:36:45.057741 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.057663    4391 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:36:45.057741 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.057670    4391 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:36:45.057831 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.057729    4391 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:36:45.057652639Z","UUID":"f5ebbdd9-8985-4aa2-b60f-1dafce9b6d77","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:36:45.058997 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:45.058978    4391 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:36:45.059103 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:45.059017    4391 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-92.us-west-2.compute.internal\" not found"
Oct 30 18:36:45.059155 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.059112    4391 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:36:45.059155 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.059133    4391 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:36:45.098399 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.098319    4391 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"]
Oct 30 18:36:45.098516 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.098436    4391 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:45.099257 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.099239    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:45.099326 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.099269    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:45.099326 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.099280    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:45.099518 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.099509    4391 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:45.099667 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.099653    4391 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:45.100211 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100194    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:45.100295 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100198    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:45.100295 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100241    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:45.100295 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100253    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:45.100295 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100216    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:45.100295 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100286    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:45.100452 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:45.100409    4391 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-92.us-west-2.compute.internal\" not found" node="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.100518 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100474    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal" event={"ID":"4ce1444a469ee2b4c13e78a90a9b0681","Type":"ContainerStarted","Data":"f1b6da25bdff6522a7ff64ea65d5b4f4f77427d58e4b0140d1495baf5771019d"}
Oct 30 18:36:45.100518 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100506    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal" event={"ID":"4ce1444a469ee2b4c13e78a90a9b0681","Type":"ContainerStarted","Data":"831b762ae9cc950299f132beb1715e116a8afc93faa442c3c7f8ea4eed440d64"}
Oct 30 18:36:45.100597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100528    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal" event={"ID":"e4b49969aba00a9d5dd95ce58685c160","Type":"ContainerStarted","Data":"3a478f7c6c1683e7a623b04a9caa25059d28d57de9e116a39cf6ce3a69df961c"}
Oct 30 18:36:45.100597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100540    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal" event={"ID":"e4b49969aba00a9d5dd95ce58685c160","Type":"ContainerDied","Data":"5a3e22e86f441d0d9181473e64dec9ecfcc223f78239b07658881d878a07bc4a"}
Oct 30 18:36:45.100597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100547    4391 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:45.100597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.100552    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal" event={"ID":"e4b49969aba00a9d5dd95ce58685c160","Type":"ContainerStarted","Data":"7671310b8bd536d1490c6752b1cfbd4d4a092141a5d767a83f57e6c0a7b0d4ee"}
Oct 30 18:36:45.101210 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.101191    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:45.101285 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.101222    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:45.101285 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.101236    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:45.101403 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:45.101389    4391 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-92.us-west-2.compute.internal\" not found" node="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.143272 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.143243    4391 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:45.143420 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.143395    4391 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:36:45.144074 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.144052    4391 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:45.145430 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.145412    4391 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:45.158055 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.158036    4391 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:45.159035 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.159016    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:45.159110 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.159049    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:45.159110 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.159060    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:45.159110 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.159095    4391 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.164290 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.164275    4391 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.192638 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.192603    4391 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:45.237068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.237040    4391 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:45.237216 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:45.237097    4391 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.237216 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.237117    4391 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.243502 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.243474    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.243621 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.243509    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ce1444a469ee2b4c13e78a90a9b0681-config\") pod \"kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"4ce1444a469ee2b4c13e78a90a9b0681\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.243621 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.243538    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.286004 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.285980    4391 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.286064 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.286050    4391 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.286622 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.286606    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeNotReady"
Oct 30 18:36:45.286699 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.286640    4391 setters.go:546] "Node became not ready" node="ip-10-0-128-92.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T18:36:45Z","lastTransitionTime":"2026-10-30T18:36:45Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 18:36:45.291498 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.291480    4391 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:45.291603 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:45.291527    4391 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.296738 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.296719    4391 nodeinfomanager.go:471] existing CSINode "ip-10-0-128-92.us-west-2.compute.internal" is owned by different node (oldNodeID="b6d23dbd-d588-4c34-b065-1175b437336c", newNodeID=""), cleaning up...
Oct 30 18:36:45.344480 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.344446    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.344651 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.344490    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.344651 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.344511    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ce1444a469ee2b4c13e78a90a9b0681-config\") pod \"kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"4ce1444a469ee2b4c13e78a90a9b0681\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.344651 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.344552    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ce1444a469ee2b4c13e78a90a9b0681-config\") pod \"kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"4ce1444a469ee2b4c13e78a90a9b0681\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.344651 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.344551    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.344651 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.344612    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4b49969aba00a9d5dd95ce58685c160-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" (UID: \"e4b49969aba00a9d5dd95ce58685c160\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:45.549571 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.549544    4391 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-92.us-west-2.compute.internal" event="NodeReady"
Oct 30 18:36:45.549741 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.549639    4391 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:36:45.950694 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:45.950658    4391 apiserver.go:52] "Watching apiserver"
Oct 30 18:36:46.040172 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.040145    4391 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:46.040313 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.040243    4391 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:46.049329 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.049298    4391 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:36:46.051314 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.051253    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-5phnp","kube-system/global-pull-secret-syncer-55w4h","openshift-dns/dns-default-xngm5","openshift-insights/insights-runtime-extractor-2454l","openshift-service-ca/service-ca-5fffc97bdf-c5q24","openshift-cluster-node-tuning-operator/tuned-rrhvc","openshift-ingress-canary/ingress-canary-clpsb","openshift-dns/node-resolver-sw4vz","kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-84s2b","kube-system/konnectivity-agent-rqbk8","openshift-ovn-kubernetes/ovnkube-node-nddlj","openshift-multus/network-metrics-daemon-4zt8w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns","openshift-multus/multus-vwxwq","openshift-network-operator/iptables-alerter-5xvkc","openshift-image-registry/node-ca-9pbdd"]
Oct 30 18:36:46.055372 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.055349    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:46.056063 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.056045    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.056378 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.056360    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.065892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.065870    4391 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:36:46.097180 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.097158    4391 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:46.097279 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:46.097233    4391 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:46.098165 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.098148    4391 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:46.098240 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:46.098194    4391 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.us-west-2.compute.internal"
Oct 30 18:36:46.102022 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102003    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:36:46.102095 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102031    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:36:46.102095 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102092    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:36:46.102207 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102165    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.102295 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102283    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.102352 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102336    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.102406 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102390    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:36:46.102458 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102423    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.102458 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102436    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:36:46.102458 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102443    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:36:46.102600 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102482    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.102600 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102488    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.102600 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102496    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:36:46.102600 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102560    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.102787 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102743    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:36:46.102787 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102753    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:36:46.102878 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102821    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h4btl\""
Oct 30 18:36:46.102878 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102842    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.102963 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102880    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.102963 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102901    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dnpxx\""
Oct 30 18:36:46.103055 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.102986    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j6hdp\""
Oct 30 18:36:46.103055 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103002    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.103055 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103012    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:36:46.103188 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103168    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.103232 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103194    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlfn7\""
Oct 30 18:36:46.103232 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103222    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:36:46.103466 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103450    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5mz5g\""
Oct 30 18:36:46.103672 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103461    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.103672 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103488    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9ph7g\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103674    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103497    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m6qzp\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103513    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vd777\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103531    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103568    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103630    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103630    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103644    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103493    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103677    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vpn2j\""
Oct 30 18:36:46.103777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.103765    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7kzh2\""
Oct 30 18:36:46.104386 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.104370    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5slhp\""
Oct 30 18:36:46.131376 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131352    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.131503 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131489    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.131556 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131522    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:36:46.131649 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131560    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.131649 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131621    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 18:36:46.131649 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131622    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxkbs\""
Oct 30 18:36:46.131789 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131661    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.131789 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131763    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vtx29\""
Oct 30 18:36:46.131875 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131817    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 18:36:46.131875 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131835    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.132090 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131966    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:36:46.132090 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.131989    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.132090 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.132037    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:36:46.132090 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.132075    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:36:46.135561 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.135541    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-swngw\""
Oct 30 18:36:46.135657 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.135549    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:36:46.135657 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.135573    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-d742r\""
Oct 30 18:36:46.135657 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.135252    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:36:46.135657 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.135613    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zmfwz\""
Oct 30 18:36:46.147998 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.147969    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.148068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148007    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-slash\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.148068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148023    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-netns\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.148068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148037    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-log-socket\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.148068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148052    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-kubernetes\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.148234 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148137    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-sys\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.148234 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148158    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6bj4\" (UniqueName: \"kubernetes.io/projected/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-kube-api-access-n6bj4\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.148234 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148194    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:46.148362 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148267    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-run\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.148362 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148290    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-lib-modules\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.148362 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148305    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-kubelet-config\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:46.148362 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148320    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-system-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.148362 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148344    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cni-binary-copy\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.148607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148387    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-bin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.148607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148483    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-daemon-config\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.148607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148520    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2601c6f1-0eb6-4038-b54c-5f8a7214db42-original-pull-secret\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:46.148607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148523    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2nlc\" (UniqueName: \"kubernetes.io/projected/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-kube-api-access-f2nlc\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.148607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148568    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovn-node-metrics-cert\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.148787 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148679    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cni-binary-copy\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.148787 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148701    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-host\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.148882 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148799    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-tmp\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.148882 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148834    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngsk\" (UniqueName: \"kubernetes.io/projected/30e470e7-fb0d-4ce4-aa97-18d038f99fae-kube-api-access-cngsk\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.148882 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148843    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-daemon-config\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.148882 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148867    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovn-node-metrics-cert\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.148882 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148872    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-tmp\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.149047 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148868    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.149047 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148910    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-netd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.149047 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148927    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.149047 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148945    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qxp\" (UniqueName: \"kubernetes.io/projected/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-kube-api-access-z6qxp\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:46.149047 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.148979    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-socket-dir-parent\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.149288 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149081    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63559645-8315-4941-b0ad-62405092b592-crio-socket\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.149288 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149114    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63559645-8315-4941-b0ad-62405092b592-data-volume\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.149288 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149163    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysconfig\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.149288 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149212    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63559645-8315-4941-b0ad-62405092b592-data-volume\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.149288 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149249    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-multus-certs\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.149288 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149275    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-registration-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.149497 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149291    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.149497 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149341    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aba3fd34-c628-4399-9f05-6e7711c3489f-cert\") pod \"ingress-canary-clpsb\" (UID: \"aba3fd34-c628-4399-9f05-6e7711c3489f\") " pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:46.149497 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149367    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-host\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:46.149497 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149416    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-multus\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.149497 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149443    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-systemd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.149497 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149476    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-bin\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.149766 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149509    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-config\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.149766 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149515    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aba3fd34-c628-4399-9f05-6e7711c3489f-cert\") pod \"ingress-canary-clpsb\" (UID: \"aba3fd34-c628-4399-9f05-6e7711c3489f\") " pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:46.149766 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149545    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30e470e7-fb0d-4ce4-aa97-18d038f99fae-config-volume\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.149766 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149607    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-system-cni-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.149766 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149700    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltx7\" (UniqueName: \"kubernetes.io/projected/65831eb9-258f-4f08-96b0-1b4556b183a7-kube-api-access-mltx7\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.149766 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149720    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30e470e7-fb0d-4ce4-aa97-18d038f99fae-config-volume\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.149766 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149735    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-config\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.149766 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149741    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c50a391f-9c33-4ece-8c52-e767af7a41b1-host-slash\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:46.149993 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149843    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9pl\" (UniqueName: \"kubernetes.io/projected/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-kube-api-access-vp9pl\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:46.149993 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149864    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.149993 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149908    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-device-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.149993 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149941    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-conf-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.149993 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149961    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-etc-kubernetes\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.149993 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.149981    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.150219 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150001    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-tmp-dir\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:46.150219 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150042    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-konnectivity-ca\") pod \"konnectivity-agent-rqbk8\" (UID: \"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9\") " pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:46.150219 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150064    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.150219 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150118    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-tmp-dir\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:46.150219 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150166    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.150448 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150230    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.150448 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150286    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c50a391f-9c33-4ece-8c52-e767af7a41b1-iptables-alerter-script\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:46.150448 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150310    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-konnectivity-ca\") pod \"konnectivity-agent-rqbk8\" (UID: \"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9\") " pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:46.150448 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150328    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.150448 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150351    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-kubelet\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.150448 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150354    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.150448 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150380    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxf4\" (UniqueName: \"kubernetes.io/projected/0d7bcd44-6514-4409-852d-18fb596acb10-kube-api-access-dfxf4\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:46.150448 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150404    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-systemd\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150488    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150521    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92v5\" (UniqueName: \"kubernetes.io/projected/12185714-c5ea-442a-9f5d-ef7af20f28c0-kube-api-access-t92v5\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150532    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c50a391f-9c33-4ece-8c52-e767af7a41b1-iptables-alerter-script\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150557    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-k8s-cni-cncf-io\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150625    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d7bcd44-6514-4409-852d-18fb596acb10-signing-cabundle\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150655    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpg6\" (UniqueName: \"kubernetes.io/projected/63559645-8315-4941-b0ad-62405092b592-kube-api-access-srpg6\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150682    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-var-lib-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150710    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150739    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-tuned\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150807    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-tuned\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.150847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150841    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-sys-fs\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150854    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d7bcd44-6514-4409-852d-18fb596acb10-signing-cabundle\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150874    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7z9\" (UniqueName: \"kubernetes.io/projected/6350fe6c-a551-41dc-88a2-f15bd2360a1c-kube-api-access-4r7z9\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150905    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-netns\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150926    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150942    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-systemd-units\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150957    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-env-overrides\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.150973    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jhk6\" (UniqueName: \"kubernetes.io/projected/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-kube-api-access-7jhk6\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151109    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-dbus\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151141    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-script-lib\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151158    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-hostroot\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151172    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d7bcd44-6514-4409-852d-18fb596acb10-signing-key\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151199    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-env-overrides\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151212    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-etc-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151247    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-ovn\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151275    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-hosts-file\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:46.151324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151316    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-agent-certs\") pod \"konnectivity-agent-rqbk8\" (UID: \"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9\") " pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151343    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d7bcd44-6514-4409-852d-18fb596acb10-signing-key\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151349    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp648\" (UniqueName: \"kubernetes.io/projected/c50a391f-9c33-4ece-8c52-e767af7a41b1-kube-api-access-xp648\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151370    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cnibin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151388    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-kubelet\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151403    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-modprobe-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151423    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-cnibin\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151448    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151477    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-socket-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151501    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6350fe6c-a551-41dc-88a2-f15bd2360a1c-ovnkube-script-lib\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151542    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-serviceca\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151550    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9-agent-certs\") pod \"konnectivity-agent-rqbk8\" (UID: \"a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9\") " pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151604    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151620    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65831eb9-258f-4f08-96b0-1b4556b183a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151697    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-node-log\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151721    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-var-lib-kubelet\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151746    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151754    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-serviceca\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151771    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151802    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-conf\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151848    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e470e7-fb0d-4ce4-aa97-18d038f99fae-tmp-dir\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151869    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpzs\" (UniqueName: \"kubernetes.io/projected/aba3fd34-c628-4399-9f05-6e7711c3489f-kube-api-access-gfpzs\") pod \"ingress-canary-clpsb\" (UID: \"aba3fd34-c628-4399-9f05-6e7711c3489f\") " pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:46.152016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.151954    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e470e7-fb0d-4ce4-aa97-18d038f99fae-tmp-dir\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.200682 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.200650    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngsk\" (UniqueName: \"kubernetes.io/projected/30e470e7-fb0d-4ce4-aa97-18d038f99fae-kube-api-access-cngsk\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.201274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.201215    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qxp\" (UniqueName: \"kubernetes.io/projected/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-kube-api-access-z6qxp\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:46.201274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.201247    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2nlc\" (UniqueName: \"kubernetes.io/projected/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-kube-api-access-f2nlc\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.201381 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.201318    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6bj4\" (UniqueName: \"kubernetes.io/projected/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-kube-api-access-n6bj4\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.202110 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.202080    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9pl\" (UniqueName: \"kubernetes.io/projected/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-kube-api-access-vp9pl\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:46.202226 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.202129    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltx7\" (UniqueName: \"kubernetes.io/projected/65831eb9-258f-4f08-96b0-1b4556b183a7-kube-api-access-mltx7\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.202542 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.202522    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92v5\" (UniqueName: \"kubernetes.io/projected/12185714-c5ea-442a-9f5d-ef7af20f28c0-kube-api-access-t92v5\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.202804 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.202784    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxf4\" (UniqueName: \"kubernetes.io/projected/0d7bcd44-6514-4409-852d-18fb596acb10-kube-api-access-dfxf4\") pod \"service-ca-5fffc97bdf-c5q24\" (UID: \"0d7bcd44-6514-4409-852d-18fb596acb10\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c5q24"
Oct 30 18:36:46.203432 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.203412    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7z9\" (UniqueName: \"kubernetes.io/projected/6350fe6c-a551-41dc-88a2-f15bd2360a1c-kube-api-access-4r7z9\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.203740 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.203725    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp648\" (UniqueName: \"kubernetes.io/projected/c50a391f-9c33-4ece-8c52-e767af7a41b1-kube-api-access-xp648\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:46.203788 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.203754    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jhk6\" (UniqueName: \"kubernetes.io/projected/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-kube-api-access-7jhk6\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:46.204045 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.204025    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpg6\" (UniqueName: \"kubernetes.io/projected/63559645-8315-4941-b0ad-62405092b592-kube-api-access-srpg6\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.204647 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.204627    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hpcs\" (UniqueName: \"kubernetes.io/projected/92bb0644-0b77-405a-bcf2-4b88b8ac4d4d-kube-api-access-4hpcs\") pod \"network-check-target-5phnp\" (UID: \"92bb0644-0b77-405a-bcf2-4b88b8ac4d4d\") " pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:46.204946 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.204932    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpzs\" (UniqueName: \"kubernetes.io/projected/aba3fd34-c628-4399-9f05-6e7711c3489f-kube-api-access-gfpzs\") pod \"ingress-canary-clpsb\" (UID: \"aba3fd34-c628-4399-9f05-6e7711c3489f\") " pod="openshift-ingress-canary/ingress-canary-clpsb"
Oct 30 18:36:46.252208 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252178    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.252342 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252216    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-netd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.252342 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252243    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.252342 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252259    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-socket-dir-parent\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.252342 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252285    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63559645-8315-4941-b0ad-62405092b592-crio-socket\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.252342 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252295    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.252342 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252308    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysconfig\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.252342 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252318    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-netd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.252342 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252334    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-multus-certs\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.252342 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252337    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252357    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-registration-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252388    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-socket-dir-parent\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252399    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysconfig\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252414    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-registration-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252411    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63559645-8315-4941-b0ad-62405092b592-crio-socket\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252414    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-multus-certs\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252453    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252533    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252562    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-host\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252603    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-multus\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252629    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-systemd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252653    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-bin\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252662    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-multus\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252622    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57a4e656-ef45-4ef7-b2f5-bbb46c7fc600-host\") pod \"node-ca-9pbdd\" (UID: \"57a4e656-ef45-4ef7-b2f5-bbb46c7fc600\") " pod="openshift-image-registry/node-ca-9pbdd"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252678    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-system-cni-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252705    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-system-cni-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252708    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-cni-bin\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.252730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252677    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-systemd\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252667    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9pbdd_57a4e656-ef45-4ef7-b2f5-bbb46c7fc600/node-ca/0.log"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252759    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c50a391f-9c33-4ece-8c52-e767af7a41b1-host-slash\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252705    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c50a391f-9c33-4ece-8c52-e767af7a41b1-host-slash\") pod \"iptables-alerter-5xvkc\" (UID: \"c50a391f-9c33-4ece-8c52-e767af7a41b1\") " pod="openshift-network-operator/iptables-alerter-5xvkc"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252816    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252835    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-device-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252850    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-conf-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252868    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-etc-kubernetes\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252891    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252923    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252927    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-device-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252938    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-etc-kubernetes\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252925    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-conf-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252968    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-kubelet\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252991    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-kubelet\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252991    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-systemd\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.252970    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253018    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-systemd\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:46.253038    4391 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253053    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253073    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-k8s-cni-cncf-io\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253113    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-var-lib-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253123    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-k8s-cni-cncf-io\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253131    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:46.253144    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:46.753121656 +0000 UTC m=+1.863854948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253149    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-var-lib-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253162    4391 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253181    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253209    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-sys-fs\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253237    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-netns\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253261    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-systemd-units\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253240    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-sys-fs\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253212    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253287    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-dbus\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:46.253274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253290    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-run-netns\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253311    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-systemd-units\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253311    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-hostroot\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253335    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-dbus\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253339    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-hostroot\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253353    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-etc-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253382    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-ovn\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253401    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-etc-openvswitch\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253404    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-hosts-file\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253428    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-run-ovn\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253429    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cnibin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253442    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07d6f30f-5f4f-4b67-9dcc-3cae33390c84-hosts-file\") pod \"node-resolver-sw4vz\" (UID: \"07d6f30f-5f4f-4b67-9dcc-3cae33390c84\") " pod="openshift-dns/node-resolver-sw4vz"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253451    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-kubelet\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253471    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-modprobe-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253476    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-cnibin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253492    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-cnibin\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253513    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-kubelet\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253521    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-socket-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253526    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65831eb9-258f-4f08-96b0-1b4556b183a7-cnibin\") pod \"multus-additional-cni-plugins-84s2b\" (UID: \"65831eb9-258f-4f08-96b0-1b4556b183a7\") " pod="openshift-multus/multus-additional-cni-plugins-84s2b"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253548    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253571    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-node-log\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253600    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12185714-c5ea-442a-9f5d-ef7af20f28c0-socket-dir\") pod \"aws-ebs-csi-driver-node-w9hns\" (UID: \"12185714-c5ea-442a-9f5d-ef7af20f28c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w9hns"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253611    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-var-lib-kubelet\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253603    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-modprobe-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253623    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-node-log\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253634    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253654    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253664    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-var-lib-kubelet\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253666    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-multus-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253670    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-conf\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253734    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253758    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-d\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253759    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-slash\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253782    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-netns\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253786    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-sysctl-conf\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253795    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-slash\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253797    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-log-socket\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253826    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-kubernetes\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253835    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-log-socket\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253858    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6350fe6c-a551-41dc-88a2-f15bd2360a1c-host-run-netns\") pod \"ovnkube-node-nddlj\" (UID: \"6350fe6c-a551-41dc-88a2-f15bd2360a1c\") " pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253853    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-sys\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253894    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-run\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253906    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-etc-kubernetes\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:46.253797    4391 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253945    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-run\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253918    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-sys\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:46.254004    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:46.753985667 +0000 UTC m=+1.864718942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.253918    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-lib-modules\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.254002    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-lib-modules\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.254044    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-kubelet-config\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.254061    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-system-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.254076    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-bin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.254097    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-host\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.254145    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-host-var-lib-cni-bin\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.254154    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b976d8d0-4b1e-43a8-874d-94d3ecde6a3c-system-cni-dir\") pod \"multus-vwxwq\" (UID: \"b976d8d0-4b1e-43a8-874d-94d3ecde6a3c\") " pod="openshift-multus/multus-vwxwq"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.254164    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2601c6f1-0eb6-4038-b54c-5f8a7214db42-kubelet-config\") pod \"global-pull-secret-syncer-55w4h\" (UID: \"2601c6f1-0eb6-4038-b54c-5f8a7214db42\") " pod="kube-system/global-pull-secret-syncer-55w4h"
Oct 30 18:36:46.254464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.254181    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5-host\") pod \"tuned-rrhvc\" (UID: \"f9c6cdf4-e8d7-4a93-9c44-98fdf90250b5\") " pod="openshift-cluster-node-tuning-operator/tuned-rrhvc"
Oct 30 18:36:46.256123 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.255423    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30e470e7-fb0d-4ce4-aa97-18d038f99fae-metrics-tls\") pod \"dns-default-xngm5\" (UID: \"30e470e7-fb0d-4ce4-aa97-18d038f99fae\") " pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.256123 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.255878    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f-metrics-certs\") pod \"network-metrics-daemon-4zt8w\" (UID: \"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f\") " pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:46.356146 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.356110    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zt8w"
Oct 30 18:36:46.357233 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.357213    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:46.729998 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.729964    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4zt8w"]
Oct 30 18:36:46.733241 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:46.733216    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfcd26b_8ca3_44fc_bfc4_2c4db9790b0f.slice/crio-c263771c75d22d57b028ad212d7a03b8953c3410f877982466b66ca85f6974f7 WatchSource:0}: Error finding container c263771c75d22d57b028ad212d7a03b8953c3410f877982466b66ca85f6974f7: Status 404 returned error can't find the container with id c263771c75d22d57b028ad212d7a03b8953c3410f877982466b66ca85f6974f7
Oct 30 18:36:46.735548 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.735532    4391 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:36:46.757794 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.757769    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.757873 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.757814    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:46.757926 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:46.757907    4391 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:46.757926 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:46.757914    4391 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:46.757985 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:46.757951    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.757939599 +0000 UTC m=+2.868672860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:46.757985 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:46.757965    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.757959355 +0000 UTC m=+2.868692616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:46.778302 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:46.778282    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xngm5"]
Oct 30 18:36:46.781426 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:46.781404    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e470e7_fb0d_4ce4_aa97_18d038f99fae.slice/crio-9ecbe79807ec34beb7a7a315db64b535637bfaa3218d946182edc149eaaafff1 WatchSource:0}: Error finding container 9ecbe79807ec34beb7a7a315db64b535637bfaa3218d946182edc149eaaafff1: Status 404 returned error can't find the container with id 9ecbe79807ec34beb7a7a315db64b535637bfaa3218d946182edc149eaaafff1
Oct 30 18:36:47.044242 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:47.044156    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xngm5" event={"ID":"30e470e7-fb0d-4ce4-aa97-18d038f99fae","Type":"ContainerStarted","Data":"9ecbe79807ec34beb7a7a315db64b535637bfaa3218d946182edc149eaaafff1"}
Oct 30 18:36:47.045388 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:47.045362    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zt8w" event={"ID":"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f","Type":"ContainerStarted","Data":"c263771c75d22d57b028ad212d7a03b8953c3410f877982466b66ca85f6974f7"}
Oct 30 18:36:47.053334 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:47.053311    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-clpsb_aba3fd34-c628-4399-9f05-6e7711c3489f/serve-healthcheck-canary/0.log"
Oct 30 18:36:47.764823 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:47.764738    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:47.764963 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:47.764823    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:47.764963 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:47.764843    4391 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:47.764963 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:47.764927    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:49.764906577 +0000 UTC m=+4.875639852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:47.764963 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:47.764939    4391 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:47.765130 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:47.764987    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:49.76497266 +0000 UTC m=+4.875705921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:48.907605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:48.907553    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:49.039262 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:49.039237    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:49.053691 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:49.053665    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xngm5" event={"ID":"30e470e7-fb0d-4ce4-aa97-18d038f99fae","Type":"ContainerStarted","Data":"cd65bb4058df08fe157922bb13749fc8a3948cc9a5d5922ed11ec1cd912920f9"}
Oct 30 18:36:49.055727 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:49.055697    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zt8w" event={"ID":"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f","Type":"ContainerStarted","Data":"2321f9041a5f2cd94e6f39dfd9e4ab47e54cacdcda704105d3c88f5c8717cebc"}
Oct 30 18:36:49.055817 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:49.055768    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zt8w" event={"ID":"cdfcd26b-8ca3-44fc-bfc4-2c4db9790b0f","Type":"ContainerStarted","Data":"0382bd8c5c19a142f8c799e82735d11eff7a77c4499814b6375eec1769764b6b"}
Oct 30 18:36:49.055884 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:49.055834    4391 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:49.780403 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:49.780369    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:49.780595 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:49.780419    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:49.780595 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:49.780495    4391 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:49.780595 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:49.780540    4391 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:49.780595 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:49.780548    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:53.78053542 +0000 UTC m=+8.891268685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:49.780719 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:36:49.780618    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls podName:63559645-8315-4941-b0ad-62405092b592 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:53.78060406 +0000 UTC m=+8.891337321 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2454l" (UID: "63559645-8315-4941-b0ad-62405092b592") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:50.061536 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:50.061449    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xngm5" event={"ID":"30e470e7-fb0d-4ce4-aa97-18d038f99fae","Type":"ContainerStarted","Data":"e8cf6c340f283ad32c416c295c0577e1854f7de60e03181a679b9db72a5525fe"}
Oct 30 18:36:50.061897 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:50.061612    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xngm5"
Oct 30 18:36:53.805910 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:53.805867    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:53.806318 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:53.805939    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:53.806526 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:53.806504    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63559645-8315-4941-b0ad-62405092b592-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:53.808234 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:53.808217    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63559645-8315-4941-b0ad-62405092b592-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2454l\" (UID: \"63559645-8315-4941-b0ad-62405092b592\") " pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:53.857356 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:53.857329    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2454l"
Oct 30 18:36:54.041008 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:54.040979    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nddlj"
Oct 30 18:36:54.302196 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:54.302159    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2454l"]
Oct 30 18:36:54.322757 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:36:54.322728    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63559645_8315_4941_b0ad_62405092b592.slice/crio-de549854b71af0efc9184f671a9a20eaad85de762fe5cfa575f8da7382fdbada WatchSource:0}: Error finding container de549854b71af0efc9184f671a9a20eaad85de762fe5cfa575f8da7382fdbada: Status 404 returned error can't find the container with id de549854b71af0efc9184f671a9a20eaad85de762fe5cfa575f8da7382fdbada
Oct 30 18:36:55.076404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:55.076372    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2454l" event={"ID":"63559645-8315-4941-b0ad-62405092b592","Type":"ContainerStarted","Data":"9ca5a1fbf5ac150772bdb029bdd0db7b5f90c83af1bd2077713291faafac9b74"}
Oct 30 18:36:55.076833 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:55.076424    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2454l" event={"ID":"63559645-8315-4941-b0ad-62405092b592","Type":"ContainerStarted","Data":"de549854b71af0efc9184f671a9a20eaad85de762fe5cfa575f8da7382fdbada"}
Oct 30 18:36:55.418144 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:55.418114    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:36:56.081188 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:56.081115    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2454l" event={"ID":"63559645-8315-4941-b0ad-62405092b592","Type":"ContainerStarted","Data":"d0e5668c3c15d893133d01be20312164a07dd9893d9abc96605fe5fb75e12ca0"}
Oct 30 18:36:58.090536 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:36:58.090493    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2454l" event={"ID":"63559645-8315-4941-b0ad-62405092b592","Type":"ContainerStarted","Data":"e1e9ad900214e87052bcd81b74f8704af47e6b986da82f7b57bcd585de014d96"}
Oct 30 18:37:00.066189 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:00.066162    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xngm5"
Oct 30 18:37:01.570761 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:01.570726    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x"]
Oct 30 18:37:01.574076 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:01.574051    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x"
Oct 30 18:37:01.626486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:01.626454    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x"]
Oct 30 18:37:01.658317 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:01.658278    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09ed0cc6-ba84-46cf-8b98-f2880f2d987b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9999x\" (UID: \"09ed0cc6-ba84-46cf-8b98-f2880f2d987b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x"
Oct 30 18:37:01.672430 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:01.672401    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 18:37:01.672430 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:01.672417    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xhtls\""
Oct 30 18:37:01.758696 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:01.758652    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09ed0cc6-ba84-46cf-8b98-f2880f2d987b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9999x\" (UID: \"09ed0cc6-ba84-46cf-8b98-f2880f2d987b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x"
Oct 30 18:37:01.761210 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:01.761179    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09ed0cc6-ba84-46cf-8b98-f2880f2d987b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9999x\" (UID: \"09ed0cc6-ba84-46cf-8b98-f2880f2d987b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x"
Oct 30 18:37:01.884269 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:01.884169    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x"
Oct 30 18:37:02.244348 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:02.244311    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x"]
Oct 30 18:37:02.248150 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:02.248121    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ed0cc6_ba84_46cf_8b98_f2880f2d987b.slice/crio-788cd41ab7396489699b5dffe4f91353acc8979270d60596c87c64574e0795b6 WatchSource:0}: Error finding container 788cd41ab7396489699b5dffe4f91353acc8979270d60596c87c64574e0795b6: Status 404 returned error can't find the container with id 788cd41ab7396489699b5dffe4f91353acc8979270d60596c87c64574e0795b6
Oct 30 18:37:03.105496 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.105459    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x" event={"ID":"09ed0cc6-ba84-46cf-8b98-f2880f2d987b","Type":"ContainerStarted","Data":"788cd41ab7396489699b5dffe4f91353acc8979270d60596c87c64574e0795b6"}
Oct 30 18:37:03.481696 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.481656    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-745d7b446d-8h9m4"]
Oct 30 18:37:03.484449 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.484414    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.570511 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.570477    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-config\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.570701 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.570524    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9db8d54f-0164-4dc1-a868-f9fc39795fc9-tmp\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.570701 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.570631    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-serving-cert\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.570701 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.570684    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-oauth-config\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.570823 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.570712    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-service-ca\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.570823 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.570737    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbfj\" (UniqueName: \"kubernetes.io/projected/9db8d54f-0164-4dc1-a868-f9fc39795fc9-kube-api-access-mmbfj\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.570920 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.570848    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-oauth-serving-cert\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.581369 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.581333    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 18:37:03.581551 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.581444    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 18:37:03.581551 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.581476    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 18:37:03.581551 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.581511    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 18:37:03.581681 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.581592    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 18:37:03.581681 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.581613    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 18:37:03.581752 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.581710    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 18:37:03.581816 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.581801    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xhdpj\""
Oct 30 18:37:03.629032 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.629003    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-745d7b446d-8h9m4"]
Oct 30 18:37:03.671389 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.671366    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-config\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.671496 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.671396    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9db8d54f-0164-4dc1-a868-f9fc39795fc9-tmp\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.671496 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.671423    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-serving-cert\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.671496 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.671447    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-oauth-config\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.671496 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.671466    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-service-ca\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.671496 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.671488    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbfj\" (UniqueName: \"kubernetes.io/projected/9db8d54f-0164-4dc1-a868-f9fc39795fc9-kube-api-access-mmbfj\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.671768 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.671518    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-oauth-serving-cert\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.672178 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.672154    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-config\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.672332 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.672315    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9db8d54f-0164-4dc1-a868-f9fc39795fc9-tmp\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.672700 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.672674    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-service-ca\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.672798 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.672776    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-oauth-serving-cert\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.674050 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.674031    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-serving-cert\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.674167 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.674137    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-oauth-config\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.730292 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.730261    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbfj\" (UniqueName: \"kubernetes.io/projected/9db8d54f-0164-4dc1-a868-f9fc39795fc9-kube-api-access-mmbfj\") pod \"console-745d7b446d-8h9m4\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") " pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:03.796429 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:03.796347    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:04.110101 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:04.110019    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x" event={"ID":"09ed0cc6-ba84-46cf-8b98-f2880f2d987b","Type":"ContainerStarted","Data":"d90951a3492abbb373ede1768da6c8084ea5fdd4a666973c614cc6a124832a53"}
Oct 30 18:37:04.110498 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:04.110250    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x"
Oct 30 18:37:04.115057 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:04.115031    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x"
Oct 30 18:37:04.156843 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:04.156816    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-745d7b446d-8h9m4"]
Oct 30 18:37:04.160496 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:04.160474    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db8d54f_0164_4dc1_a868_f9fc39795fc9.slice/crio-007b8b483ff0db44c81a0922f4e3c42536f508ed0eba9e2cde4f1b67feb6b5e2 WatchSource:0}: Error finding container 007b8b483ff0db44c81a0922f4e3c42536f508ed0eba9e2cde4f1b67feb6b5e2: Status 404 returned error can't find the container with id 007b8b483ff0db44c81a0922f4e3c42536f508ed0eba9e2cde4f1b67feb6b5e2
Oct 30 18:37:04.217817 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:04.217758    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9999x" podStartSLOduration=1.830848051 podStartE2EDuration="3.217744779s" totalImagesPullingTime="1.386896728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:04.217396212 +0000 UTC m=+19.328129489" watchObservedRunningTime="2026-10-30 18:37:04.217744779 +0000 UTC m=+19.328478062"
Oct 30 18:37:05.117227 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:05.117186    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745d7b446d-8h9m4" event={"ID":"9db8d54f-0164-4dc1-a868-f9fc39795fc9","Type":"ContainerStarted","Data":"007b8b483ff0db44c81a0922f4e3c42536f508ed0eba9e2cde4f1b67feb6b5e2"}
Oct 30 18:37:08.127485 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:08.127435    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745d7b446d-8h9m4" event={"ID":"9db8d54f-0164-4dc1-a868-f9fc39795fc9","Type":"ContainerStarted","Data":"3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98"}
Oct 30 18:37:08.237479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:08.237427    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-745d7b446d-8h9m4" podStartSLOduration=2.3006622119999998 podStartE2EDuration="5.237413466s" totalImagesPullingTime="2.936751254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:08.237134895 +0000 UTC m=+23.347868177" watchObservedRunningTime="2026-10-30 18:37:08.237413466 +0000 UTC m=+23.348146745"
Oct 30 18:37:11.250761 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:37:11.250721    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:37:11.748982 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.748943    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"]
Oct 30 18:37:11.751916 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.751895    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.823107 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.823070    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pcj5\" (UniqueName: \"kubernetes.io/projected/80d7ed04-4512-4e3e-80e4-b880915f0d9b-kube-api-access-8pcj5\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.823277 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.823111    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80d7ed04-4512-4e3e-80e4-b880915f0d9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.823277 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.823138    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80d7ed04-4512-4e3e-80e4-b880915f0d9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.823277 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.823262    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80d7ed04-4512-4e3e-80e4-b880915f0d9b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.848505 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.848474    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5kd7r\""
Oct 30 18:37:11.848505 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.848498    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 18:37:11.848728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.848491    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 18:37:11.848728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.848649    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:37:11.848728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.848651    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:37:11.848728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.848708    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:37:11.849409 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.849386    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"]
Oct 30 18:37:11.857159 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.857140    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cmz4n"]
Oct 30 18:37:11.859403 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.859385    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:11.895918 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.895897    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"]
Oct 30 18:37:11.898392 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.898376    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:11.908972 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.908954    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:37:11.909078 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.909033    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:37:11.909233 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.909220    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:37:11.909296 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.909282    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zw6ts\""
Oct 30 18:37:11.924631 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.924606    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:11.924729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.924640    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80d7ed04-4512-4e3e-80e4-b880915f0d9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.924729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.924661    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e2c5a8a-5c17-497f-a042-268d9481f8f1-metrics-client-ca\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:11.924842 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.924806    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e2c5a8a-5c17-497f-a042-268d9481f8f1-root\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:11.924842 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.924829    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:11.924952 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.924849    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80d7ed04-4512-4e3e-80e4-b880915f0d9b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.924952 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.924917    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-tls\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:11.924952 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.924943    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e2c5a8a-5c17-497f-a042-268d9481f8f1-sys\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:11.925092 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.924964    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:11.925092 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925023    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80d7ed04-4512-4e3e-80e4-b880915f0d9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.925092 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925068    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:11.925192 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925106    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mp9\" (UniqueName: \"kubernetes.io/projected/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-api-access-66mp9\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:11.925192 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925155    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-wtmp\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:11.925269 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925200    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhfxj\" (UniqueName: \"kubernetes.io/projected/7e2c5a8a-5c17-497f-a042-268d9481f8f1-kube-api-access-hhfxj\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:11.925269 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925236    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:11.925344 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925267    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:11.925344 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925299    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-textfile\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:11.925344 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925323    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:11.925461 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925372    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pcj5\" (UniqueName: \"kubernetes.io/projected/80d7ed04-4512-4e3e-80e4-b880915f0d9b-kube-api-access-8pcj5\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.925495 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.925466    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80d7ed04-4512-4e3e-80e4-b880915f0d9b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.927102 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.927083    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80d7ed04-4512-4e3e-80e4-b880915f0d9b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.927172 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.927154    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80d7ed04-4512-4e3e-80e4-b880915f0d9b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:11.953589 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.953554    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"]
Oct 30 18:37:11.979463 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:11.979440    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pcj5\" (UniqueName: \"kubernetes.io/projected/80d7ed04-4512-4e3e-80e4-b880915f0d9b-kube-api-access-8pcj5\") pod \"openshift-state-metrics-7bd7489957-jmrdq\" (UID: \"80d7ed04-4512-4e3e-80e4-b880915f0d9b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:12.009381 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.009325    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 18:37:12.009464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.009427    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 18:37:12.009520 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.009505    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l94js\""
Oct 30 18:37:12.009874 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.009861    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 18:37:12.025690 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025671    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.025690 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025698    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-textfile\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.025856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025716    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.025856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025737    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.025856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025754    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e2c5a8a-5c17-497f-a042-268d9481f8f1-metrics-client-ca\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.025856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025768    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e2c5a8a-5c17-497f-a042-268d9481f8f1-root\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.025856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025792    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.025856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025821    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-tls\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.025856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025836    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e2c5a8a-5c17-497f-a042-268d9481f8f1-root\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.025856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025851    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e2c5a8a-5c17-497f-a042-268d9481f8f1-sys\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.026230 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025885    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.026230 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025926    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.026230 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025956    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mp9\" (UniqueName: \"kubernetes.io/projected/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-api-access-66mp9\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.026230 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.025981    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-wtmp\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.026230 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.026010    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhfxj\" (UniqueName: \"kubernetes.io/projected/7e2c5a8a-5c17-497f-a042-268d9481f8f1-kube-api-access-hhfxj\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.026230 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.026037    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.026518 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.026434    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.026807 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.026612    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-wtmp\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.026807 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.026615    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.026807 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.026649    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e2c5a8a-5c17-497f-a042-268d9481f8f1-sys\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.026807 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.026780    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-textfile\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.026981 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.026807    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e2c5a8a-5c17-497f-a042-268d9481f8f1-metrics-client-ca\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.026981 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.026970    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.027304 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.027283    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.028617 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.028597    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-tls\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.028736 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.028718    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.028845 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.028820    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e2c5a8a-5c17-497f-a042-268d9481f8f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.029204 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.029186    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.064942 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.064916    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"
Oct 30 18:37:12.080353 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.080319    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhfxj\" (UniqueName: \"kubernetes.io/projected/7e2c5a8a-5c17-497f-a042-268d9481f8f1-kube-api-access-hhfxj\") pod \"node-exporter-cmz4n\" (UID: \"7e2c5a8a-5c17-497f-a042-268d9481f8f1\") " pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.080552 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.080533    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mp9\" (UniqueName: \"kubernetes.io/projected/0ee5578e-70b9-45ad-b720-7d8bd3a2bda9-kube-api-access-66mp9\") pod \"kube-state-metrics-5dc8d7d89b-fpmkc\" (UID: \"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.171651 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.171614    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cmz4n"
Oct 30 18:37:12.181281 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:12.181252    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2c5a8a_5c17_497f_a042_268d9481f8f1.slice/crio-5cf57849158c5206fd53823e44afae10a46e27ffeac68b6c72902b4c597505dc WatchSource:0}: Error finding container 5cf57849158c5206fd53823e44afae10a46e27ffeac68b6c72902b4c597505dc: Status 404 returned error can't find the container with id 5cf57849158c5206fd53823e44afae10a46e27ffeac68b6c72902b4c597505dc
Oct 30 18:37:12.211213 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.211184    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"
Oct 30 18:37:12.476845 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.476810    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq"]
Oct 30 18:37:12.480092 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:12.480068    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d7ed04_4512_4e3e_80e4_b880915f0d9b.slice/crio-59b97079895544d0cad8d7385eb9bd9d45f2eb8ca3b430902214ad71810a1b9a WatchSource:0}: Error finding container 59b97079895544d0cad8d7385eb9bd9d45f2eb8ca3b430902214ad71810a1b9a: Status 404 returned error can't find the container with id 59b97079895544d0cad8d7385eb9bd9d45f2eb8ca3b430902214ad71810a1b9a
Oct 30 18:37:12.572165 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:12.572137    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc"]
Oct 30 18:37:12.575105 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:12.575078    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee5578e_70b9_45ad_b720_7d8bd3a2bda9.slice/crio-0c5e89e8643333c4e16e24e20a880fdf30de81bb13ddcd4eedab3cbf1a08ea4b WatchSource:0}: Error finding container 0c5e89e8643333c4e16e24e20a880fdf30de81bb13ddcd4eedab3cbf1a08ea4b: Status 404 returned error can't find the container with id 0c5e89e8643333c4e16e24e20a880fdf30de81bb13ddcd4eedab3cbf1a08ea4b
Oct 30 18:37:13.146418 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:13.146376    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmz4n" event={"ID":"7e2c5a8a-5c17-497f-a042-268d9481f8f1","Type":"ContainerStarted","Data":"5cf57849158c5206fd53823e44afae10a46e27ffeac68b6c72902b4c597505dc"}
Oct 30 18:37:13.147774 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:13.147736    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc" event={"ID":"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9","Type":"ContainerStarted","Data":"0c5e89e8643333c4e16e24e20a880fdf30de81bb13ddcd4eedab3cbf1a08ea4b"}
Oct 30 18:37:13.149906 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:13.149879    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq" event={"ID":"80d7ed04-4512-4e3e-80e4-b880915f0d9b","Type":"ContainerStarted","Data":"262c6838188ee291489acb65faddf66a230749de798e9a9ebf32967ac920b6e0"}
Oct 30 18:37:13.150065 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:13.149933    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq" event={"ID":"80d7ed04-4512-4e3e-80e4-b880915f0d9b","Type":"ContainerStarted","Data":"46739ce2b87cac3bd511d7d27339b409f699213960649a477646a6757306df8e"}
Oct 30 18:37:13.150065 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:13.149964    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq" event={"ID":"80d7ed04-4512-4e3e-80e4-b880915f0d9b","Type":"ContainerStarted","Data":"59b97079895544d0cad8d7385eb9bd9d45f2eb8ca3b430902214ad71810a1b9a"}
Oct 30 18:37:13.797185 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:13.797109    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:13.797185 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:13.797178    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:13.803418 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:13.803394    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:14.094741 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.094667    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rqbk8"
Oct 30 18:37:14.155658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.155621    4391 generic.go:356] "Generic (PLEG): container finished" podID="7e2c5a8a-5c17-497f-a042-268d9481f8f1" containerID="6b5b7431455b60a1db4fdf7677ca9ad51fb88b539a68490eebcf71e5758fea9c" exitCode=0
Oct 30 18:37:14.156496 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.156467    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmz4n" event={"ID":"7e2c5a8a-5c17-497f-a042-268d9481f8f1","Type":"ContainerDied","Data":"6b5b7431455b60a1db4fdf7677ca9ad51fb88b539a68490eebcf71e5758fea9c"}
Oct 30 18:37:14.161528 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.161505    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:14.255722 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.255696    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:37:14.258423 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.258403    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.303728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.303703    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:37:14.307644 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.307621    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 18:37:14.307734 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.307646    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 18:37:14.307734 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.307623    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jgbc8\""
Oct 30 18:37:14.315572 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.315547    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 18:37:14.315691 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.315592    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 18:37:14.315691 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.315650    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 18:37:14.316094 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.316073    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346039    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-tls-assets\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346084    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-config-out\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346147    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346185    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346226    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346257    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346284    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-web-config\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346306    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346367    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346391    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346412    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnpkx\" (UniqueName: \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-kube-api-access-wnpkx\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346437    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-config-volume\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.353486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.346461    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.354467 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.354422    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 18:37:14.354877 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.354847    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 18:37:14.363019 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.362982    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 18:37:14.391641 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.391616    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67469644d8-ljdzh"]
Oct 30 18:37:14.393785 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.393761    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.446306 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.446284    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 18:37:14.446859 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.446838    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-tls-assets\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.446966 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.446867    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/327c49f2-91b9-499f-9d69-076092cb28c2-tmp\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.446966 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.446884    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-config-out\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.446966 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.446908    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.446966 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.446932    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.447141 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.446982    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-trusted-ca-bundle\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.447141 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.447031    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-service-ca\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.447665 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.447644    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.448168 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.448148    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-oauth-serving-cert\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.448356 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.448337    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.449270 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449250    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-serving-cert\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.449394 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.447957    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.449498 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449418    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-web-config\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.449498 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449457    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.449671 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449508    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-oauth-config\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.449671 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449548    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.449671 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449576    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.449671 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449632    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnpkx\" (UniqueName: \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-kube-api-access-wnpkx\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.449671 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449656    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-console-config\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.449944 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449681    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-config-volume\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.449944 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449709    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.449944 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449734    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54vp\" (UniqueName: \"kubernetes.io/projected/327c49f2-91b9-499f-9d69-076092cb28c2-kube-api-access-c54vp\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.449944 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.449785    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-tls-assets\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.451067 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.450770    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.451694 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.451673    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-config-out\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.451818 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.451675    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.453065 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.453010    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67469644d8-ljdzh"]
Oct 30 18:37:14.454070 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.454009    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-web-config\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.454313 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.454165    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.455204 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.455181    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.456016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.455993    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-config-volume\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.456209 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.456185    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.456413 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.456374    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.463014 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.462988    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.478642 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.478610    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"]
Oct 30 18:37:14.483104 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.483083    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.506906 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.506886    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnpkx\" (UniqueName: \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-kube-api-access-wnpkx\") pod \"alertmanager-main-0\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.534706 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.534681    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 18:37:14.534975 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.534952    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 18:37:14.535077 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.535016    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aiiti476qqhlr\""
Oct 30 18:37:14.535077 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.535019    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ngkvj\""
Oct 30 18:37:14.535077 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.534959    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 18:37:14.535077 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.535023    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 18:37:14.535219 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.535081    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 18:37:14.542114 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.542088    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"]
Oct 30 18:37:14.550834 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.550812    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/327c49f2-91b9-499f-9d69-076092cb28c2-tmp\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.550957 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.550847    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.550957 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.550867    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-trusted-ca-bundle\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.550957 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.550897    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-service-ca\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.551122 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.550983    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9th\" (UniqueName: \"kubernetes.io/projected/3f08cc73-2de9-403d-bf75-afecaf1d8b63-kube-api-access-vl9th\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.551122 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551028    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.551122 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551064    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-oauth-serving-cert\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.551274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551128    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-serving-cert\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.551274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551162    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-tls\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.551274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551184    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f08cc73-2de9-403d-bf75-afecaf1d8b63-metrics-client-ca\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.551274 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551211    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-grpc-tls\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.551473 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551276    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/327c49f2-91b9-499f-9d69-076092cb28c2-tmp\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.551473 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551324    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-oauth-config\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.551473 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551375    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-console-config\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.551473 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551406    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.551473 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551438    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54vp\" (UniqueName: \"kubernetes.io/projected/327c49f2-91b9-499f-9d69-076092cb28c2-kube-api-access-c54vp\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.551888 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551863    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-service-ca\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.551985 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.551888    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-oauth-serving-cert\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.552119 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.552102    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-trusted-ca-bundle\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.552250 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.552112    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-console-config\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.552332 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.552119    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.552332 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.552306    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f08cc73-2de9-403d-bf75-afecaf1d8b63-tmp\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.553649 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.553631    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-serving-cert\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.553868 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.553846    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-oauth-config\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.592943 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.592914    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:37:14.606119 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.606090    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54vp\" (UniqueName: \"kubernetes.io/projected/327c49f2-91b9-499f-9d69-076092cb28c2-kube-api-access-c54vp\") pod \"console-67469644d8-ljdzh\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") " pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.655894 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.655833    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.656059 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.655918    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.656483 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.656391    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f08cc73-2de9-403d-bf75-afecaf1d8b63-tmp\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.656483 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.656446    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.657022 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.656752    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9th\" (UniqueName: \"kubernetes.io/projected/3f08cc73-2de9-403d-bf75-afecaf1d8b63-kube-api-access-vl9th\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.657022 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.656797    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.657022 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.656837    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-tls\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.657022 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.656862    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f08cc73-2de9-403d-bf75-afecaf1d8b63-metrics-client-ca\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.657022 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.656889    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-grpc-tls\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.657313 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.657184    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f08cc73-2de9-403d-bf75-afecaf1d8b63-tmp\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.658732 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.658681    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.659331 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.659312    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f08cc73-2de9-403d-bf75-afecaf1d8b63-metrics-client-ca\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.659720 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.659694    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.660756 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.660712    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.661073 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.661023    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.661170 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.661161    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-grpc-tls\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.662877 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.662853    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3f08cc73-2de9-403d-bf75-afecaf1d8b63-secret-thanos-querier-tls\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.710410 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.710373    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:14.711265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.711240    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9th\" (UniqueName: \"kubernetes.io/projected/3f08cc73-2de9-403d-bf75-afecaf1d8b63-kube-api-access-vl9th\") pod \"thanos-querier-55c7bb6cbd-wnchg\" (UID: \"3f08cc73-2de9-403d-bf75-afecaf1d8b63\") " pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:14.808867 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:14.808834    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:15.017364 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.017328    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:37:15.022139 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:15.022106    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680e4ff2_b452_472a_9f27_96515de0a671.slice/crio-cd8335406c4e347885f1ef1cefe24baad9ed66f49c43619fa1433cd3e901bfec WatchSource:0}: Error finding container cd8335406c4e347885f1ef1cefe24baad9ed66f49c43619fa1433cd3e901bfec: Status 404 returned error can't find the container with id cd8335406c4e347885f1ef1cefe24baad9ed66f49c43619fa1433cd3e901bfec
Oct 30 18:37:15.079166 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.079139    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67469644d8-ljdzh"]
Oct 30 18:37:15.085114 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:15.085086    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327c49f2_91b9_499f_9d69_076092cb28c2.slice/crio-340b99828f0cd90f346904b9520589717e308a1cf20ed2f31377a711f8779c7b WatchSource:0}: Error finding container 340b99828f0cd90f346904b9520589717e308a1cf20ed2f31377a711f8779c7b: Status 404 returned error can't find the container with id 340b99828f0cd90f346904b9520589717e308a1cf20ed2f31377a711f8779c7b
Oct 30 18:37:15.159725 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.159691    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerStarted","Data":"cd8335406c4e347885f1ef1cefe24baad9ed66f49c43619fa1433cd3e901bfec"}
Oct 30 18:37:15.161910 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.161883    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc" event={"ID":"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9","Type":"ContainerStarted","Data":"95b0c322f34c76a4bbab9fb01c784ad18e6fbe0818ca734ccf8fb08beb63954f"}
Oct 30 18:37:15.162024 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.161947    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc" event={"ID":"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9","Type":"ContainerStarted","Data":"0629a66bff57e7447ff11983c9c9a547b7929c3f83c113bac556ddc82c9ba8d6"}
Oct 30 18:37:15.162024 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.161970    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc" event={"ID":"0ee5578e-70b9-45ad-b720-7d8bd3a2bda9","Type":"ContainerStarted","Data":"5f63620a10d264ec04c06841c43b4dbd57fe2daac57a50a2e96f0fb3a266aaa9"}
Oct 30 18:37:15.163998 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.163972    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq" event={"ID":"80d7ed04-4512-4e3e-80e4-b880915f0d9b","Type":"ContainerStarted","Data":"2e63d90a10d67e585a2a4c93c5651d82c2707c083a2adf8264e3df5805e0905f"}
Oct 30 18:37:15.165242 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.165222    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67469644d8-ljdzh" event={"ID":"327c49f2-91b9-499f-9d69-076092cb28c2","Type":"ContainerStarted","Data":"340b99828f0cd90f346904b9520589717e308a1cf20ed2f31377a711f8779c7b"}
Oct 30 18:37:15.167087 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.167064    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmz4n" event={"ID":"7e2c5a8a-5c17-497f-a042-268d9481f8f1","Type":"ContainerStarted","Data":"273b9c28d0cce5b8ab974b23496a0998d5e2ed49ce884b9e74ad3cae76f8433f"}
Oct 30 18:37:15.167180 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.167102    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmz4n" event={"ID":"7e2c5a8a-5c17-497f-a042-268d9481f8f1","Type":"ContainerStarted","Data":"60ac2bba8ddfccb74c793d7d91e3253900afedd654a6d510c1bbd944c2a3c58b"}
Oct 30 18:37:15.210325 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.210291    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"]
Oct 30 18:37:15.213915 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:15.213881    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f08cc73_2de9_403d_bf75_afecaf1d8b63.slice/crio-1e322177deecb0fcf5b8853cdff0fedd2c0792858302472ecc1b3570116a1ac6 WatchSource:0}: Error finding container 1e322177deecb0fcf5b8853cdff0fedd2c0792858302472ecc1b3570116a1ac6: Status 404 returned error can't find the container with id 1e322177deecb0fcf5b8853cdff0fedd2c0792858302472ecc1b3570116a1ac6
Oct 30 18:37:15.272512 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.272456    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fpmkc" podStartSLOduration=2.518444689 podStartE2EDuration="4.272442827s" totalImagesPullingTime="1.753998138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:15.27130057 +0000 UTC m=+30.382033853" watchObservedRunningTime="2026-10-30 18:37:15.272442827 +0000 UTC m=+30.383176109"
Oct 30 18:37:15.382796 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.382735    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cmz4n" podStartSLOduration=2.933545569 podStartE2EDuration="4.382721425s" totalImagesPullingTime="1.36560238s" totalInitContainerRuntime="83.573476ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:15.382479574 +0000 UTC m=+30.493212861" watchObservedRunningTime="2026-10-30 18:37:15.382721425 +0000 UTC m=+30.493454707"
Oct 30 18:37:15.420462 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.420432    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5phnp"
Oct 30 18:37:15.492082 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:15.492021    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jmrdq" podStartSLOduration=2.807932477 podStartE2EDuration="4.492000235s" totalImagesPullingTime="1.684067758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:15.491482605 +0000 UTC m=+30.602215887" watchObservedRunningTime="2026-10-30 18:37:15.492000235 +0000 UTC m=+30.602733514"
Oct 30 18:37:16.171948 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.171902    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg" event={"ID":"3f08cc73-2de9-403d-bf75-afecaf1d8b63","Type":"ContainerStarted","Data":"1e322177deecb0fcf5b8853cdff0fedd2c0792858302472ecc1b3570116a1ac6"}
Oct 30 18:37:16.173924 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.173894    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67469644d8-ljdzh" event={"ID":"327c49f2-91b9-499f-9d69-076092cb28c2","Type":"ContainerStarted","Data":"949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606"}
Oct 30 18:37:16.285870 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.285811    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67469644d8-ljdzh" podStartSLOduration=2.285794101 podStartE2EDuration="2.285794101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:16.285745282 +0000 UTC m=+31.396478569" watchObservedRunningTime="2026-10-30 18:37:16.285794101 +0000 UTC m=+31.396527380"
Oct 30 18:37:16.587806 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.587724    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"]
Oct 30 18:37:16.618147 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.618111    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"
Oct 30 18:37:16.645600 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.645555    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"]
Oct 30 18:37:16.668974 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.668807    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 18:37:16.668974 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.668934    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4zm6v\""
Oct 30 18:37:16.673486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.673458    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzhp\" (UniqueName: \"kubernetes.io/projected/cbb0e89e-34f4-4ee5-adca-8f086abc024c-kube-api-access-2zzhp\") pod \"monitoring-plugin-77f4f76594-hz5k9\" (UID: \"cbb0e89e-34f4-4ee5-adca-8f086abc024c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"
Oct 30 18:37:16.673632 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.673548    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbb0e89e-34f4-4ee5-adca-8f086abc024c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hz5k9\" (UID: \"cbb0e89e-34f4-4ee5-adca-8f086abc024c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"
Oct 30 18:37:16.683841 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.683818    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-56d957c68c-qcj58"]
Oct 30 18:37:16.700282 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.700259    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.742657 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.742605    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56d957c68c-qcj58"]
Oct 30 18:37:16.749694 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.749671    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 18:37:16.749836 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.749702    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 18:37:16.770088 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.770065    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wgvqf\""
Oct 30 18:37:16.770088 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.770074    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 18:37:16.770278 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.770088    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 18:37:16.770278 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.770161    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dfaib6rmtvhcs\""
Oct 30 18:37:16.774474 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.774448    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzhp\" (UniqueName: \"kubernetes.io/projected/cbb0e89e-34f4-4ee5-adca-8f086abc024c-kube-api-access-2zzhp\") pod \"monitoring-plugin-77f4f76594-hz5k9\" (UID: \"cbb0e89e-34f4-4ee5-adca-8f086abc024c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"
Oct 30 18:37:16.774612 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.774591    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ca3276a-5fe8-4978-97b2-79098bb956d3-audit-log\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.774685 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.774650    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ca3276a-5fe8-4978-97b2-79098bb956d3-secret-metrics-server-tls\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.774776 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.774755    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ca3276a-5fe8-4978-97b2-79098bb956d3-client-ca-bundle\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.774835 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.774808    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca3276a-5fe8-4978-97b2-79098bb956d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.774914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.774895    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbb0e89e-34f4-4ee5-adca-8f086abc024c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hz5k9\" (UID: \"cbb0e89e-34f4-4ee5-adca-8f086abc024c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"
Oct 30 18:37:16.774963 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.774934    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ca3276a-5fe8-4978-97b2-79098bb956d3-metrics-server-audit-profiles\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.775012 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.774959    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m424p\" (UniqueName: \"kubernetes.io/projected/2ca3276a-5fe8-4978-97b2-79098bb956d3-kube-api-access-m424p\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.775058 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.775026    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ca3276a-5fe8-4978-97b2-79098bb956d3-secret-metrics-server-client-certs\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.777530 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.777502    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbb0e89e-34f4-4ee5-adca-8f086abc024c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hz5k9\" (UID: \"cbb0e89e-34f4-4ee5-adca-8f086abc024c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"
Oct 30 18:37:16.828950 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.828923    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzhp\" (UniqueName: \"kubernetes.io/projected/cbb0e89e-34f4-4ee5-adca-8f086abc024c-kube-api-access-2zzhp\") pod \"monitoring-plugin-77f4f76594-hz5k9\" (UID: \"cbb0e89e-34f4-4ee5-adca-8f086abc024c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"
Oct 30 18:37:16.876110 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.876034    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca3276a-5fe8-4978-97b2-79098bb956d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.876110 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.876086    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ca3276a-5fe8-4978-97b2-79098bb956d3-metrics-server-audit-profiles\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.876282 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.876110    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m424p\" (UniqueName: \"kubernetes.io/projected/2ca3276a-5fe8-4978-97b2-79098bb956d3-kube-api-access-m424p\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.876282 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.876147    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ca3276a-5fe8-4978-97b2-79098bb956d3-secret-metrics-server-client-certs\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.876282 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.876221    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ca3276a-5fe8-4978-97b2-79098bb956d3-audit-log\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.876282 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.876260    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ca3276a-5fe8-4978-97b2-79098bb956d3-secret-metrics-server-tls\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.876418 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.876302    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ca3276a-5fe8-4978-97b2-79098bb956d3-client-ca-bundle\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.876851 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.876800    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2ca3276a-5fe8-4978-97b2-79098bb956d3-audit-log\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.876960 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.876873    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ca3276a-5fe8-4978-97b2-79098bb956d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.877210 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.877186    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2ca3276a-5fe8-4978-97b2-79098bb956d3-metrics-server-audit-profiles\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.879033 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.879012    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2ca3276a-5fe8-4978-97b2-79098bb956d3-client-ca-bundle\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.879326 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.879302    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2ca3276a-5fe8-4978-97b2-79098bb956d3-secret-metrics-server-client-certs\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.879326 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.879314    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2ca3276a-5fe8-4978-97b2-79098bb956d3-secret-metrics-server-tls\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.930158 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.930127    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m424p\" (UniqueName: \"kubernetes.io/projected/2ca3276a-5fe8-4978-97b2-79098bb956d3-kube-api-access-m424p\") pod \"metrics-server-56d957c68c-qcj58\" (UID: \"2ca3276a-5fe8-4978-97b2-79098bb956d3\") " pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:16.943110 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:16.943089    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"
Oct 30 18:37:17.031525 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.031492    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:17.509154 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.509119    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5cf66459f-jhs5g"]
Oct 30 18:37:17.531714 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.531682    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.578678 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.578649    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"]
Oct 30 18:37:17.582176 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.582133    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 18:37:17.582303 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:17.582138    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb0e89e_34f4_4ee5_adca_8f086abc024c.slice/crio-3453d5349b5150defa34fd9960b39ea17ef6933068ddd5c19240be13dab0e95d WatchSource:0}: Error finding container 3453d5349b5150defa34fd9960b39ea17ef6933068ddd5c19240be13dab0e95d: Status 404 returned error can't find the container with id 3453d5349b5150defa34fd9960b39ea17ef6933068ddd5c19240be13dab0e95d
Oct 30 18:37:17.582420 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.582372    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56d957c68c-qcj58"]
Oct 30 18:37:17.583186 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.583164    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 18:37:17.583459 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.583445    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-telemeter-client-tls\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.583500 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.583476    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-secret-telemeter-client\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.583500 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.583495    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l28f\" (UniqueName: \"kubernetes.io/projected/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-kube-api-access-4l28f\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.583567 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.583511    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-serving-certs-ca-bundle\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.583567 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.583532    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.583671 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.583600    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-metrics-client-ca\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.583732 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.583716    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-federate-client-tls\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.583773 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.583754    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.587285 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:17.587265    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca3276a_5fe8_4978_97b2_79098bb956d3.slice/crio-ec264109c19d33d3d26a2895c1a50cadc8d28985ad012f4ab6c7d7bb406b9f97 WatchSource:0}: Error finding container ec264109c19d33d3d26a2895c1a50cadc8d28985ad012f4ab6c7d7bb406b9f97: Status 404 returned error can't find the container with id ec264109c19d33d3d26a2895c1a50cadc8d28985ad012f4ab6c7d7bb406b9f97
Oct 30 18:37:17.625488 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.625456    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cf66459f-jhs5g"]
Oct 30 18:37:17.656507 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.655944    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 18:37:17.656507 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.656027    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 18:37:17.656507 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.656152    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jks46\""
Oct 30 18:37:17.657603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.657518    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 18:37:17.657603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.657558    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 18:37:17.685119 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.685087    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.685286 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.685128    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-metrics-client-ca\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.685286 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.685174    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-federate-client-tls\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.685286 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.685199    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.685286 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.685236    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-telemeter-client-tls\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.685286 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.685258    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-secret-telemeter-client\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.685286 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.685287    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l28f\" (UniqueName: \"kubernetes.io/projected/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-kube-api-access-4l28f\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.685570 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.685310    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-serving-certs-ca-bundle\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.686978 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.686005    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.686978 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.686012    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-serving-certs-ca-bundle\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.686978 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.686538    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-metrics-client-ca\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.689151 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.689128    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-telemeter-client-tls\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.689269 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.689136    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-federate-client-tls\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.689523 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.689482    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.690658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.690619    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-secret-telemeter-client\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.741891 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.741860    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l28f\" (UniqueName: \"kubernetes.io/projected/195b50cc-7fbc-451e-b2c7-cea82d04d2f8-kube-api-access-4l28f\") pod \"telemeter-client-5cf66459f-jhs5g\" (UID: \"195b50cc-7fbc-451e-b2c7-cea82d04d2f8\") " pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:17.882616 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:17.882525    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g"
Oct 30 18:37:18.195463 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:18.195427    4391 generic.go:356] "Generic (PLEG): container finished" podID="680e4ff2-b452-472a-9f27-96515de0a671" containerID="0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e" exitCode=0
Oct 30 18:37:18.195634 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:18.195497    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerDied","Data":"0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e"}
Oct 30 18:37:18.196836 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:18.196815    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9" event={"ID":"cbb0e89e-34f4-4ee5-adca-8f086abc024c","Type":"ContainerStarted","Data":"3453d5349b5150defa34fd9960b39ea17ef6933068ddd5c19240be13dab0e95d"}
Oct 30 18:37:18.197988 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:18.197967    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56d957c68c-qcj58" event={"ID":"2ca3276a-5fe8-4978-97b2-79098bb956d3","Type":"ContainerStarted","Data":"ec264109c19d33d3d26a2895c1a50cadc8d28985ad012f4ab6c7d7bb406b9f97"}
Oct 30 18:37:18.309077 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:18.309053    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cf66459f-jhs5g"]
Oct 30 18:37:19.205365 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:19.205312    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g" event={"ID":"195b50cc-7fbc-451e-b2c7-cea82d04d2f8","Type":"ContainerStarted","Data":"5fa24f6bbf29c59e3336628628076dd251b9375951250599643b375c2f6f581f"}
Oct 30 18:37:19.208385 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:19.208356    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg" event={"ID":"3f08cc73-2de9-403d-bf75-afecaf1d8b63","Type":"ContainerStarted","Data":"264b15a060e394dea868db2185256ac8dd2f4dc15e45e8becc844caa01e113cd"}
Oct 30 18:37:19.208512 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:19.208427    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg" event={"ID":"3f08cc73-2de9-403d-bf75-afecaf1d8b63","Type":"ContainerStarted","Data":"341043644ba6a83011ac30853856b7951f8e3f0351ae5449fb229c9591ce8a7c"}
Oct 30 18:37:19.208512 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:19.208454    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg" event={"ID":"3f08cc73-2de9-403d-bf75-afecaf1d8b63","Type":"ContainerStarted","Data":"17a3a8c558b5b5ed898d51375d1d16b0a1335d6e87db75b3cdc12d64bc935e0e"}
Oct 30 18:37:20.214507 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:20.214469    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56d957c68c-qcj58" event={"ID":"2ca3276a-5fe8-4978-97b2-79098bb956d3","Type":"ContainerStarted","Data":"7a39fd6dd836109968abb8c9c24713a2dcbd0ff8fc1906a86eb093286be47e69"}
Oct 30 18:37:20.324782 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:20.324674    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-56d957c68c-qcj58" podStartSLOduration=1.879511895 podStartE2EDuration="4.324656512s" totalImagesPullingTime="2.445144617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:20.323779668 +0000 UTC m=+35.434512950" watchObservedRunningTime="2026-10-30 18:37:20.324656512 +0000 UTC m=+35.435389798"
Oct 30 18:37:21.220739 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:21.220706    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg" event={"ID":"3f08cc73-2de9-403d-bf75-afecaf1d8b63","Type":"ContainerStarted","Data":"c2702659e55741df1695a75bf984142762873c9470532fff30614c7eaaac130c"}
Oct 30 18:37:21.222780 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:21.222756    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerStarted","Data":"2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b"}
Oct 30 18:37:21.224441 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:21.224414    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9" event={"ID":"cbb0e89e-34f4-4ee5-adca-8f086abc024c","Type":"ContainerStarted","Data":"e36d9400c1371d2e671a12ca5fec342eb84f09787eff245592c9a7051b98fe95"}
Oct 30 18:37:21.350637 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:21.349495    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9" podStartSLOduration=2.5387226529999998 podStartE2EDuration="5.349480133s" totalImagesPullingTime="2.81075748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:21.34884068 +0000 UTC m=+36.459573965" watchObservedRunningTime="2026-10-30 18:37:21.349480133 +0000 UTC m=+36.460213406"
Oct 30 18:37:22.231877 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.231842    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerStarted","Data":"4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58"}
Oct 30 18:37:22.232344 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.231916    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerStarted","Data":"09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6"}
Oct 30 18:37:22.232344 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.231958    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerStarted","Data":"e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e"}
Oct 30 18:37:22.232344 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.231995    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerStarted","Data":"ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6"}
Oct 30 18:37:22.232344 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.232019    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerStarted","Data":"0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19"}
Oct 30 18:37:22.234041 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.234014    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g" event={"ID":"195b50cc-7fbc-451e-b2c7-cea82d04d2f8","Type":"ContainerStarted","Data":"60de14b8738afd627ac317cd13c7b25182cec0fdcb97aec2fe4f7ffff4f7af6f"}
Oct 30 18:37:22.234164 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.234069    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g" event={"ID":"195b50cc-7fbc-451e-b2c7-cea82d04d2f8","Type":"ContainerStarted","Data":"0fe098bfa255a8c140daa798248e04476ba0c6d32f9264c98c9ab8e8a6bbb0c5"}
Oct 30 18:37:22.234164 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.234098    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g" event={"ID":"195b50cc-7fbc-451e-b2c7-cea82d04d2f8","Type":"ContainerStarted","Data":"8ceb07769c9614087587ab2dc4fdbfd4cf54db8ee186085aa0fcbd8c4f2a8c01"}
Oct 30 18:37:22.237071 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.237047    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg" event={"ID":"3f08cc73-2de9-403d-bf75-afecaf1d8b63","Type":"ContainerStarted","Data":"8e9a61db25e7a79e3691c832ba65b4e28f10a78b3a4a4da9c59b6cd880f42675"}
Oct 30 18:37:22.237177 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.237101    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg" event={"ID":"3f08cc73-2de9-403d-bf75-afecaf1d8b63","Type":"ContainerStarted","Data":"569b83bc38ab1b486a980b17023ade685cd657f8c980cd9638a97a72bcf4ba61"}
Oct 30 18:37:22.237177 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.237134    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"
Oct 30 18:37:22.242403 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.242382    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hz5k9"
Oct 30 18:37:22.359720 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.359666    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.493614366 podStartE2EDuration="8.359651749s" totalImagesPullingTime="4.845462461s" totalInitContainerRuntime="20.574922ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:22.357439074 +0000 UTC m=+37.468172357" watchObservedRunningTime="2026-10-30 18:37:22.359651749 +0000 UTC m=+37.470385031"
Oct 30 18:37:22.581220 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.581113    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5cf66459f-jhs5g" podStartSLOduration=2.7175039549999997 podStartE2EDuration="5.581094137s" totalImagesPullingTime="2.863590182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:22.580184141 +0000 UTC m=+37.690917424" watchObservedRunningTime="2026-10-30 18:37:22.581094137 +0000 UTC m=+37.691827423"
Oct 30 18:37:22.699352 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:22.699296    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg" podStartSLOduration=3.251130272 podStartE2EDuration="8.699279615s" totalImagesPullingTime="5.448149343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:22.697905163 +0000 UTC m=+37.808638445" watchObservedRunningTime="2026-10-30 18:37:22.699279615 +0000 UTC m=+37.810012894"
Oct 30 18:37:23.241441 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.241407    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:23.247479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.247452    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55c7bb6cbd-wnchg"
Oct 30 18:37:23.465791 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.465759    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67469644d8-ljdzh"]
Oct 30 18:37:23.496573 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.496502    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8547bbb8db-lfnll"]
Oct 30 18:37:23.499818 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.499797    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.541376 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.541352    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-oauth-config\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.541509 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.541380    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-serving-cert\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.541509 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.541399    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-service-ca\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.541624 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.541522    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb969ca-4b98-4926-95ea-6190bd56c62e-tmp\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.541624 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.541544    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4vpd\" (UniqueName: \"kubernetes.io/projected/7fb969ca-4b98-4926-95ea-6190bd56c62e-kube-api-access-f4vpd\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.541624 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.541564    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-trusted-ca-bundle\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.541780 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.541696    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-config\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.541833 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.541782    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-oauth-serving-cert\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.557493 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.557472    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8547bbb8db-lfnll"]
Oct 30 18:37:23.642703 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.642668    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb969ca-4b98-4926-95ea-6190bd56c62e-tmp\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.642703 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.642707    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4vpd\" (UniqueName: \"kubernetes.io/projected/7fb969ca-4b98-4926-95ea-6190bd56c62e-kube-api-access-f4vpd\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.642927 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.642815    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-trusted-ca-bundle\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.642927 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.642870    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-config\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.643041 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.642926    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-oauth-serving-cert\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.643041 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.642962    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-oauth-config\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.643041 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.643018    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-serving-cert\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.643192 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.643049    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-service-ca\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.643192 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.643052    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb969ca-4b98-4926-95ea-6190bd56c62e-tmp\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.643758 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.643736    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-trusted-ca-bundle\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.643758 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.643753    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-config\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.643897 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.643766    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-oauth-serving-cert\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.643897 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.643836    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-service-ca\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.645469 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.645437    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-oauth-config\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.645469 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.645458    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-serving-cert\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.699418 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.699386    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4vpd\" (UniqueName: \"kubernetes.io/projected/7fb969ca-4b98-4926-95ea-6190bd56c62e-kube-api-access-f4vpd\") pod \"console-8547bbb8db-lfnll\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") " pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:23.810958 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:23.810887    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:24.175883 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:24.175853    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8547bbb8db-lfnll"]
Oct 30 18:37:24.179268 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:37:24.179239    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb969ca_4b98_4926_95ea_6190bd56c62e.slice/crio-6396df837d1928cc78545e24a1a3bc91ba5eddbc416003df5c6c982b5824e828 WatchSource:0}: Error finding container 6396df837d1928cc78545e24a1a3bc91ba5eddbc416003df5c6c982b5824e828: Status 404 returned error can't find the container with id 6396df837d1928cc78545e24a1a3bc91ba5eddbc416003df5c6c982b5824e828
Oct 30 18:37:24.247490 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:24.247457    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8547bbb8db-lfnll" event={"ID":"7fb969ca-4b98-4926-95ea-6190bd56c62e","Type":"ContainerStarted","Data":"bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17"}
Oct 30 18:37:24.247922 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:24.247509    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8547bbb8db-lfnll" event={"ID":"7fb969ca-4b98-4926-95ea-6190bd56c62e","Type":"ContainerStarted","Data":"6396df837d1928cc78545e24a1a3bc91ba5eddbc416003df5c6c982b5824e828"}
Oct 30 18:37:24.360707 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:24.360661    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8547bbb8db-lfnll" podStartSLOduration=1.360648968 podStartE2EDuration="1.360648968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:24.359113806 +0000 UTC m=+39.469847090" watchObservedRunningTime="2026-10-30 18:37:24.360648968 +0000 UTC m=+39.471382250"
Oct 30 18:37:24.710735 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:24.710701    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:33.811517 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:33.811432    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:33.811944 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:33.811537    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:33.815950 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:33.815929    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:34.288205 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:34.288178    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:37:34.447365 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:34.447333    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-745d7b446d-8h9m4"]
Oct 30 18:37:37.031711 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:37.031670    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:37.032078 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:37.031729    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:41.251410 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:37:41.251372    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:37:48.485672 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.485609    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67469644d8-ljdzh" podUID="327c49f2-91b9-499f-9d69-076092cb28c2" containerName="console" containerID="cri-o://949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606" gracePeriod=15
Oct 30 18:37:48.736020 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.735964    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:48.777694 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.777659    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327c49f2-91b9-499f-9d69-076092cb28c2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/327c49f2-91b9-499f-9d69-076092cb28c2-tmp\") pod \"327c49f2-91b9-499f-9d69-076092cb28c2\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") "
Oct 30 18:37:48.777694 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.777696    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-oauth-config\") pod \"327c49f2-91b9-499f-9d69-076092cb28c2\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") "
Oct 30 18:37:48.777900 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.777735    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-service-ca\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-service-ca\") pod \"327c49f2-91b9-499f-9d69-076092cb28c2\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") "
Oct 30 18:37:48.777900 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.777763    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-trusted-ca-bundle\") pod \"327c49f2-91b9-499f-9d69-076092cb28c2\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") "
Oct 30 18:37:48.777900 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.777780    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327c49f2-91b9-499f-9d69-076092cb28c2-kube-api-access-c54vp\" (UniqueName: \"kubernetes.io/projected/327c49f2-91b9-499f-9d69-076092cb28c2-kube-api-access-c54vp\") pod \"327c49f2-91b9-499f-9d69-076092cb28c2\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") "
Oct 30 18:37:48.777900 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.777820    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-console-config\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-console-config\") pod \"327c49f2-91b9-499f-9d69-076092cb28c2\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") "
Oct 30 18:37:48.777900 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.777885    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-serving-cert\") pod \"327c49f2-91b9-499f-9d69-076092cb28c2\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") "
Oct 30 18:37:48.778145 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.777922    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-oauth-serving-cert\") pod \"327c49f2-91b9-499f-9d69-076092cb28c2\" (UID: \"327c49f2-91b9-499f-9d69-076092cb28c2\") "
Oct 30 18:37:48.778145 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.778029    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327c49f2-91b9-499f-9d69-076092cb28c2-tmp" pod "327c49f2-91b9-499f-9d69-076092cb28c2" (UID: "327c49f2-91b9-499f-9d69-076092cb28c2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:48.778255 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.778164    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-service-ca" pod "327c49f2-91b9-499f-9d69-076092cb28c2" (UID: "327c49f2-91b9-499f-9d69-076092cb28c2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:48.778255 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.778209    4391 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/327c49f2-91b9-499f-9d69-076092cb28c2-tmp\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:48.778361 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.778267    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-trusted-ca-bundle" pod "327c49f2-91b9-499f-9d69-076092cb28c2" (UID: "327c49f2-91b9-499f-9d69-076092cb28c2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:48.778440 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.778422    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-oauth-serving-cert" pod "327c49f2-91b9-499f-9d69-076092cb28c2" (UID: "327c49f2-91b9-499f-9d69-076092cb28c2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:48.778567 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.778542    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-console-config" pod "327c49f2-91b9-499f-9d69-076092cb28c2" (UID: "327c49f2-91b9-499f-9d69-076092cb28c2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:48.780164 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.780134    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-serving-cert" pod "327c49f2-91b9-499f-9d69-076092cb28c2" (UID: "327c49f2-91b9-499f-9d69-076092cb28c2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:48.780258 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.780169    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-oauth-config" pod "327c49f2-91b9-499f-9d69-076092cb28c2" (UID: "327c49f2-91b9-499f-9d69-076092cb28c2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:48.780258 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.780181    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327c49f2-91b9-499f-9d69-076092cb28c2-kube-api-access-c54vp" pod "327c49f2-91b9-499f-9d69-076092cb28c2" (UID: "327c49f2-91b9-499f-9d69-076092cb28c2"). InnerVolumeSpecName "kube-api-access-c54vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:48.878713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.878673    4391 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-oauth-serving-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:48.878713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.878705    4391 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-oauth-config\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:48.878713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.878716    4391 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:48.878933 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.878725    4391 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-trusted-ca-bundle\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:48.878933 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.878734    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54vp\" (UniqueName: \"kubernetes.io/projected/327c49f2-91b9-499f-9d69-076092cb28c2-kube-api-access-c54vp\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:48.878933 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.878743    4391 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/327c49f2-91b9-499f-9d69-076092cb28c2-console-config\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:48.878933 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:48.878752    4391 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/327c49f2-91b9-499f-9d69-076092cb28c2-console-serving-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:49.342286 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:49.342200    4391 generic.go:356] "Generic (PLEG): container finished" podID="327c49f2-91b9-499f-9d69-076092cb28c2" containerID="949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606" exitCode=0
Oct 30 18:37:49.342286 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:49.342253    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67469644d8-ljdzh" event={"ID":"327c49f2-91b9-499f-9d69-076092cb28c2","Type":"ContainerDied","Data":"949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606"}
Oct 30 18:37:49.342286 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:49.342266    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67469644d8-ljdzh"
Oct 30 18:37:49.342545 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:49.342306    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67469644d8-ljdzh" event={"ID":"327c49f2-91b9-499f-9d69-076092cb28c2","Type":"ContainerDied","Data":"340b99828f0cd90f346904b9520589717e308a1cf20ed2f31377a711f8779c7b"}
Oct 30 18:37:49.342545 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:49.342331    4391 scope.go:122] "RemoveContainer" containerID="949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606"
Oct 30 18:37:49.350126 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:49.350110    4391 scope.go:122] "RemoveContainer" containerID="949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606"
Oct 30 18:37:49.350413 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:37:49.350388    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606\": container with ID starting with 949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606 not found: ID does not exist" containerID="949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606"
Oct 30 18:37:49.350468 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:49.350412    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606"} err="failed to get container status \"949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606\": rpc error: code = NotFound desc = could not find container \"949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606\": container with ID starting with 949794f754e7f7087a06a4449c516b57240b7ee957b411d29298d8cbba10a606 not found: ID does not exist"
Oct 30 18:37:49.498762 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:49.498719    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67469644d8-ljdzh"]
Oct 30 18:37:49.545438 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:49.545397    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67469644d8-ljdzh"]
Oct 30 18:37:51.003864 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:51.003831    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327c49f2-91b9-499f-9d69-076092cb28c2" path="/var/lib/kubelet/pods/327c49f2-91b9-499f-9d69-076092cb28c2/volumes"
Oct 30 18:37:57.037725 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:57.037696    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:57.041304 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:57.041284    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-56d957c68c-qcj58"
Oct 30 18:37:59.467487 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.467444    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-745d7b446d-8h9m4" podUID="9db8d54f-0164-4dc1-a868-f9fc39795fc9" containerName="console" containerID="cri-o://3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98" gracePeriod=15
Oct 30 18:37:59.712596 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.712556    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:37:59.775402 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.775316    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-serving-cert\") pod \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") "
Oct 30 18:37:59.775402 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.775374    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-oauth-config\") pod \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") "
Oct 30 18:37:59.775402 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.775397    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db8d54f-0164-4dc1-a868-f9fc39795fc9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9db8d54f-0164-4dc1-a868-f9fc39795fc9-tmp\") pod \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") "
Oct 30 18:37:59.775728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.775424    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-config\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-config\") pod \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") "
Oct 30 18:37:59.775728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.775444    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-service-ca\") pod \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") "
Oct 30 18:37:59.775728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.775469    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db8d54f-0164-4dc1-a868-f9fc39795fc9-kube-api-access-mmbfj\" (UniqueName: \"kubernetes.io/projected/9db8d54f-0164-4dc1-a868-f9fc39795fc9-kube-api-access-mmbfj\") pod \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") "
Oct 30 18:37:59.775728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.775489    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-oauth-serving-cert\") pod \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\" (UID: \"9db8d54f-0164-4dc1-a868-f9fc39795fc9\") "
Oct 30 18:37:59.775904 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.775784    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db8d54f-0164-4dc1-a868-f9fc39795fc9-tmp" pod "9db8d54f-0164-4dc1-a868-f9fc39795fc9" (UID: "9db8d54f-0164-4dc1-a868-f9fc39795fc9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:59.775965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.775942    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-config" pod "9db8d54f-0164-4dc1-a868-f9fc39795fc9" (UID: "9db8d54f-0164-4dc1-a868-f9fc39795fc9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:59.775965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.775956    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-service-ca" pod "9db8d54f-0164-4dc1-a868-f9fc39795fc9" (UID: "9db8d54f-0164-4dc1-a868-f9fc39795fc9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:59.776085 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.776053    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-oauth-serving-cert" pod "9db8d54f-0164-4dc1-a868-f9fc39795fc9" (UID: "9db8d54f-0164-4dc1-a868-f9fc39795fc9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:59.777576 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.777555    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-serving-cert" pod "9db8d54f-0164-4dc1-a868-f9fc39795fc9" (UID: "9db8d54f-0164-4dc1-a868-f9fc39795fc9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:59.777709 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.777688    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-oauth-config" pod "9db8d54f-0164-4dc1-a868-f9fc39795fc9" (UID: "9db8d54f-0164-4dc1-a868-f9fc39795fc9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:59.777817 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.777794    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db8d54f-0164-4dc1-a868-f9fc39795fc9-kube-api-access-mmbfj" pod "9db8d54f-0164-4dc1-a868-f9fc39795fc9" (UID: "9db8d54f-0164-4dc1-a868-f9fc39795fc9"). InnerVolumeSpecName "kube-api-access-mmbfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:59.876444 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.876407    4391 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-serving-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.876444 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.876434    4391 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-oauth-config\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.876444 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.876444    4391 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9db8d54f-0164-4dc1-a868-f9fc39795fc9-tmp\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.876444 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.876453    4391 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-console-config\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.876710 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.876462    4391 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.876710 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.876471    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbfj\" (UniqueName: \"kubernetes.io/projected/9db8d54f-0164-4dc1-a868-f9fc39795fc9-kube-api-access-mmbfj\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:59.876710 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:37:59.876482    4391 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db8d54f-0164-4dc1-a868-f9fc39795fc9-oauth-serving-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:00.384307 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:00.384275    4391 generic.go:356] "Generic (PLEG): container finished" podID="9db8d54f-0164-4dc1-a868-f9fc39795fc9" containerID="3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98" exitCode=0
Oct 30 18:38:00.384528 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:00.384325    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-745d7b446d-8h9m4"
Oct 30 18:38:00.384528 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:00.384365    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745d7b446d-8h9m4" event={"ID":"9db8d54f-0164-4dc1-a868-f9fc39795fc9","Type":"ContainerDied","Data":"3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98"}
Oct 30 18:38:00.384528 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:00.384425    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-745d7b446d-8h9m4" event={"ID":"9db8d54f-0164-4dc1-a868-f9fc39795fc9","Type":"ContainerDied","Data":"007b8b483ff0db44c81a0922f4e3c42536f508ed0eba9e2cde4f1b67feb6b5e2"}
Oct 30 18:38:00.384528 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:00.384451    4391 scope.go:122] "RemoveContainer" containerID="3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98"
Oct 30 18:38:00.393141 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:00.393125    4391 scope.go:122] "RemoveContainer" containerID="3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98"
Oct 30 18:38:00.393382 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:00.393362    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98\": container with ID starting with 3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98 not found: ID does not exist" containerID="3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98"
Oct 30 18:38:00.393445 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:00.393395    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98"} err="failed to get container status \"3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98\": rpc error: code = NotFound desc = could not find container \"3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98\": container with ID starting with 3147a35ce07842c06dae40248cd615abeb98b78c94ed781d31836cc316d9bf98 not found: ID does not exist"
Oct 30 18:38:00.539346 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:00.539317    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-745d7b446d-8h9m4"]
Oct 30 18:38:00.586017 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:00.585984    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-745d7b446d-8h9m4"]
Oct 30 18:38:01.004007 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.003975    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db8d54f-0164-4dc1-a868-f9fc39795fc9" path="/var/lib/kubelet/pods/9db8d54f-0164-4dc1-a868-f9fc39795fc9/volumes"
Oct 30 18:38:01.712994 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.712961    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57968cdc5f-b5vgr"]
Oct 30 18:38:01.713398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.713289    4391 cpu_manager.go:414] "containerMap: removing container" podUID="9db8d54f-0164-4dc1-a868-f9fc39795fc9" containerName="console"
Oct 30 18:38:01.713398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.713300    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db8d54f-0164-4dc1-a868-f9fc39795fc9" containerName="console"
Oct 30 18:38:01.713398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.713313    4391 cpu_manager.go:414] "containerMap: removing container" podUID="327c49f2-91b9-499f-9d69-076092cb28c2" containerName="console"
Oct 30 18:38:01.713398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.713319    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327c49f2-91b9-499f-9d69-076092cb28c2" containerName="console"
Oct 30 18:38:01.713398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.713378    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="327c49f2-91b9-499f-9d69-076092cb28c2" containerName="console"
Oct 30 18:38:01.713398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.713392    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db8d54f-0164-4dc1-a868-f9fc39795fc9" containerName="console"
Oct 30 18:38:01.717395 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.717378    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.792380 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.792342    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-console-config\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.792380 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.792380    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-service-ca\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.792686 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.792529    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae7355b-e6dc-410d-95f8-075179bd184f-console-serving-cert\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.792686 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.792603    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-trusted-ca-bundle\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.792686 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.792657    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-oauth-serving-cert\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.792841 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.792689    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w9jz\" (UniqueName: \"kubernetes.io/projected/eae7355b-e6dc-410d-95f8-075179bd184f-kube-api-access-7w9jz\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.792841 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.792815    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eae7355b-e6dc-410d-95f8-075179bd184f-tmp\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.792923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.792842    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae7355b-e6dc-410d-95f8-075179bd184f-console-oauth-config\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.809072 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.809013    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57968cdc5f-b5vgr"]
Oct 30 18:38:01.893224 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.893189    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eae7355b-e6dc-410d-95f8-075179bd184f-tmp\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.893224 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.893227    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae7355b-e6dc-410d-95f8-075179bd184f-console-oauth-config\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.893475 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.893258    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-console-config\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.893475 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.893279    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-service-ca\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.893475 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.893323    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae7355b-e6dc-410d-95f8-075179bd184f-console-serving-cert\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.893475 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.893345    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-trusted-ca-bundle\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.893475 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.893371    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-oauth-serving-cert\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.893822 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.893520    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w9jz\" (UniqueName: \"kubernetes.io/projected/eae7355b-e6dc-410d-95f8-075179bd184f-kube-api-access-7w9jz\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.893822 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.893646    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eae7355b-e6dc-410d-95f8-075179bd184f-tmp\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.894092 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.894071    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-oauth-serving-cert\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.894210 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.894187    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-service-ca\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.894284 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.894260    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-trusted-ca-bundle\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.894369 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.894352    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae7355b-e6dc-410d-95f8-075179bd184f-console-config\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.895764 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.895746    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae7355b-e6dc-410d-95f8-075179bd184f-console-oauth-config\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.895882 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.895864    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae7355b-e6dc-410d-95f8-075179bd184f-console-serving-cert\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:01.948959 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:01.948930    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w9jz\" (UniqueName: \"kubernetes.io/projected/eae7355b-e6dc-410d-95f8-075179bd184f-kube-api-access-7w9jz\") pod \"console-57968cdc5f-b5vgr\" (UID: \"eae7355b-e6dc-410d-95f8-075179bd184f\") " pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:02.027528 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:02.027434    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:02.392248 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:02.392224    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57968cdc5f-b5vgr"]
Oct 30 18:38:02.394656 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:38:02.394624    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae7355b_e6dc_410d_95f8_075179bd184f.slice/crio-88b524f9cf22a1e8266d95fa1a07d55ed44f4fb61abfeb3ee9b3613050dd15bf WatchSource:0}: Error finding container 88b524f9cf22a1e8266d95fa1a07d55ed44f4fb61abfeb3ee9b3613050dd15bf: Status 404 returned error can't find the container with id 88b524f9cf22a1e8266d95fa1a07d55ed44f4fb61abfeb3ee9b3613050dd15bf
Oct 30 18:38:03.398244 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:03.398208    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57968cdc5f-b5vgr" event={"ID":"eae7355b-e6dc-410d-95f8-075179bd184f","Type":"ContainerStarted","Data":"6b4e39d27ec158c1caf85a9b436c5df2d4b539d223d25b95c2e9a7b23fc160a9"}
Oct 30 18:38:03.398652 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:03.398259    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57968cdc5f-b5vgr" event={"ID":"eae7355b-e6dc-410d-95f8-075179bd184f","Type":"ContainerStarted","Data":"88b524f9cf22a1e8266d95fa1a07d55ed44f4fb61abfeb3ee9b3613050dd15bf"}
Oct 30 18:38:03.509864 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:03.509818    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57968cdc5f-b5vgr" podStartSLOduration=2.509804877 podStartE2EDuration="2.509804877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:03.508752096 +0000 UTC m=+78.619485379" watchObservedRunningTime="2026-10-30 18:38:03.509804877 +0000 UTC m=+78.620538159"
Oct 30 18:38:08.486441 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:08.486405    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:08.486923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:08.486699    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="alertmanager" containerID="cri-o://2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b" gracePeriod=120
Oct 30 18:38:08.486923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:08.486764    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy-web" containerID="cri-o://ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6" gracePeriod=120
Oct 30 18:38:08.486923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:08.486766    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy-metric" containerID="cri-o://09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6" gracePeriod=120
Oct 30 18:38:08.486923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:08.486799    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="config-reloader" containerID="cri-o://0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19" gracePeriod=120
Oct 30 18:38:08.486923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:08.486839    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy" containerID="cri-o://e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e" gracePeriod=120
Oct 30 18:38:08.486923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:08.486889    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="prom-label-proxy" containerID="cri-o://4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58" gracePeriod=120
Oct 30 18:38:09.425459 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.425430    4391 generic.go:356] "Generic (PLEG): container finished" podID="680e4ff2-b452-472a-9f27-96515de0a671" containerID="4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58" exitCode=0
Oct 30 18:38:09.425459 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.425455    4391 generic.go:356] "Generic (PLEG): container finished" podID="680e4ff2-b452-472a-9f27-96515de0a671" containerID="09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6" exitCode=0
Oct 30 18:38:09.425459 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.425463    4391 generic.go:356] "Generic (PLEG): container finished" podID="680e4ff2-b452-472a-9f27-96515de0a671" containerID="e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e" exitCode=0
Oct 30 18:38:09.425714 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.425470    4391 generic.go:356] "Generic (PLEG): container finished" podID="680e4ff2-b452-472a-9f27-96515de0a671" containerID="0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19" exitCode=0
Oct 30 18:38:09.425714 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.425475    4391 generic.go:356] "Generic (PLEG): container finished" podID="680e4ff2-b452-472a-9f27-96515de0a671" containerID="2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b" exitCode=0
Oct 30 18:38:09.425714 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.425508    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerDied","Data":"4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58"}
Oct 30 18:38:09.425714 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.425597    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerDied","Data":"09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6"}
Oct 30 18:38:09.425714 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.425628    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerDied","Data":"e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e"}
Oct 30 18:38:09.425714 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.425651    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerDied","Data":"0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19"}
Oct 30 18:38:09.425714 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.425675    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerDied","Data":"2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b"}
Oct 30 18:38:09.719421 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.719391    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:09.855361 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855325    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-cluster-tls-config\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.855361 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855368    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-metric\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.855648 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855390    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-web\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.855648 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855439    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-main-tls\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.855648 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855461    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-kube-api-access-wnpkx\" (UniqueName: \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-kube-api-access-wnpkx\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.855648 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855482    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-main-db\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.855648 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855509    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-trusted-ca-bundle\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.855648 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855553    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-tls-assets\" (UniqueName: \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-tls-assets\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.855648 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855604    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.855648 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855641    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-config-out\" (UniqueName: \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-config-out\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.856064 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855685    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-metrics-client-ca\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.856064 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855713    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-web-config\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-web-config\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.856064 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855758    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-config-volume\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-config-volume\") pod \"680e4ff2-b452-472a-9f27-96515de0a671\" (UID: \"680e4ff2-b452-472a-9f27-96515de0a671\") "
Oct 30 18:38:09.856064 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.855998    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-trusted-ca-bundle" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:09.856264 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.856144    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-main-db" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:38:09.856828 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.856538    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-metrics-client-ca" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:09.858848 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.858423    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-metric" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:09.858848 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.858795    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-main-tls" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:09.858848 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.858810    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-web" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:09.859064 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.858893    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-kube-api-access-wnpkx" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "kube-api-access-wnpkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:09.859308 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.859273    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-config-volume" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:09.859389 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.859350    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-tls-assets" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:09.859477 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.859458    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-config-out" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:38:09.859842 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.859825    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:09.862736 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.862713    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-cluster-tls-config" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:09.869815 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.869792    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-web-config" pod "680e4ff2-b452-472a-9f27-96515de0a671" (UID: "680e4ff2-b452-472a-9f27-96515de0a671"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:09.956637 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956527    4391 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-config-volume\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956637 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956561    4391 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-cluster-tls-config\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956637 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956576    4391 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956637 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956613    4391 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956637 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956629    4391 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-main-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956644    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnpkx\" (UniqueName: \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-kube-api-access-wnpkx\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956658    4391 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-main-db\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956670    4391 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956684    4391 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/680e4ff2-b452-472a-9f27-96515de0a671-tls-assets\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956698    4391 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956711    4391 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/680e4ff2-b452-472a-9f27-96515de0a671-config-out\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956725    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680e4ff2-b452-472a-9f27-96515de0a671-metrics-client-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:09.956923 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:09.956737    4391 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/680e4ff2-b452-472a-9f27-96515de0a671-web-config\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:10.432566 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.432529    4391 generic.go:356] "Generic (PLEG): container finished" podID="680e4ff2-b452-472a-9f27-96515de0a671" containerID="ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6" exitCode=0
Oct 30 18:38:10.432758 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.432599    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerDied","Data":"ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6"}
Oct 30 18:38:10.432758 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.432640    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.432758 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.432675    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"680e4ff2-b452-472a-9f27-96515de0a671","Type":"ContainerDied","Data":"cd8335406c4e347885f1ef1cefe24baad9ed66f49c43619fa1433cd3e901bfec"}
Oct 30 18:38:10.432758 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.432721    4391 scope.go:122] "RemoveContainer" containerID="4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58"
Oct 30 18:38:10.440277 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.440245    4391 scope.go:122] "RemoveContainer" containerID="09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6"
Oct 30 18:38:10.447084 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.447065    4391 scope.go:122] "RemoveContainer" containerID="e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e"
Oct 30 18:38:10.453264 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.453245    4391 scope.go:122] "RemoveContainer" containerID="ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6"
Oct 30 18:38:10.459730 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.459707    4391 scope.go:122] "RemoveContainer" containerID="0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19"
Oct 30 18:38:10.466398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.466377    4391 scope.go:122] "RemoveContainer" containerID="2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b"
Oct 30 18:38:10.477428 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.477398    4391 scope.go:122] "RemoveContainer" containerID="0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e"
Oct 30 18:38:10.484438 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.484419    4391 scope.go:122] "RemoveContainer" containerID="4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58"
Oct 30 18:38:10.484698 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:10.484679    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58\": container with ID starting with 4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58 not found: ID does not exist" containerID="4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58"
Oct 30 18:38:10.484748 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.484708    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58"} err="failed to get container status \"4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58\": rpc error: code = NotFound desc = could not find container \"4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58\": container with ID starting with 4c0549a9ef01843ee2bfec5ab0dc19b1403495bcc6abc655035274f8a43f3c58 not found: ID does not exist"
Oct 30 18:38:10.484748 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.484726    4391 scope.go:122] "RemoveContainer" containerID="09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6"
Oct 30 18:38:10.484937 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:10.484919    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6\": container with ID starting with 09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6 not found: ID does not exist" containerID="09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6"
Oct 30 18:38:10.484973 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.484942    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6"} err="failed to get container status \"09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6\": rpc error: code = NotFound desc = could not find container \"09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6\": container with ID starting with 09cdcb5f188a2190ac7ee4500374a0fd66165cdbc6cfdc206c5ed0d1353d73c6 not found: ID does not exist"
Oct 30 18:38:10.484973 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.484954    4391 scope.go:122] "RemoveContainer" containerID="e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e"
Oct 30 18:38:10.485151 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:10.485135    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e\": container with ID starting with e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e not found: ID does not exist" containerID="e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e"
Oct 30 18:38:10.485198 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.485155    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e"} err="failed to get container status \"e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e\": rpc error: code = NotFound desc = could not find container \"e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e\": container with ID starting with e3fc9d7718a4fe29cb7ad83137ae455eb0a3f23affaadf7ebc505fa85339bd7e not found: ID does not exist"
Oct 30 18:38:10.485198 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.485166    4391 scope.go:122] "RemoveContainer" containerID="ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6"
Oct 30 18:38:10.485350 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:10.485335    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6\": container with ID starting with ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6 not found: ID does not exist" containerID="ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6"
Oct 30 18:38:10.485384 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.485353    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6"} err="failed to get container status \"ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6\": rpc error: code = NotFound desc = could not find container \"ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6\": container with ID starting with ef69da77502e986e7ca573e5d942617d038352997a9b035291718d3a1529d3e6 not found: ID does not exist"
Oct 30 18:38:10.485384 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.485364    4391 scope.go:122] "RemoveContainer" containerID="0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19"
Oct 30 18:38:10.485552 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:10.485538    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19\": container with ID starting with 0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19 not found: ID does not exist" containerID="0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19"
Oct 30 18:38:10.485610 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.485555    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19"} err="failed to get container status \"0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19\": rpc error: code = NotFound desc = could not find container \"0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19\": container with ID starting with 0edb85d2aab85e7c2b8fb4ec82854ebefbf888622d032c606aef765aa3c72f19 not found: ID does not exist"
Oct 30 18:38:10.485610 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.485566    4391 scope.go:122] "RemoveContainer" containerID="2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b"
Oct 30 18:38:10.485772 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:10.485757    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b\": container with ID starting with 2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b not found: ID does not exist" containerID="2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b"
Oct 30 18:38:10.485810 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.485775    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b"} err="failed to get container status \"2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b\": rpc error: code = NotFound desc = could not find container \"2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b\": container with ID starting with 2b220fe6745fac8caefcaa949d99959d65573bcc697ab38b467e2e48d8dbc55b not found: ID does not exist"
Oct 30 18:38:10.485810 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.485787    4391 scope.go:122] "RemoveContainer" containerID="0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e"
Oct 30 18:38:10.485973 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:10.485958    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e\": container with ID starting with 0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e not found: ID does not exist" containerID="0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e"
Oct 30 18:38:10.486007 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.485975    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e"} err="failed to get container status \"0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e\": rpc error: code = NotFound desc = could not find container \"0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e\": container with ID starting with 0a8e1902429b61a7ee150805be0826c9b0ee0a4c207533ddef16de6e58d2586e not found: ID does not exist"
Oct 30 18:38:10.638782 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.638750    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:10.685517 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.685444    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:10.686360 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686341    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:10.686680 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686667    4391 cpu_manager.go:414] "containerMap: removing container" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="config-reloader"
Oct 30 18:38:10.686713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686682    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="config-reloader"
Oct 30 18:38:10.686713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686696    4391 cpu_manager.go:414] "containerMap: removing container" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="init-config-reloader"
Oct 30 18:38:10.686713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686701    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="init-config-reloader"
Oct 30 18:38:10.686713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686708    4391 cpu_manager.go:414] "containerMap: removing container" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="prom-label-proxy"
Oct 30 18:38:10.686713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686714    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="prom-label-proxy"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686725    4391 cpu_manager.go:414] "containerMap: removing container" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="alertmanager"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686729    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="alertmanager"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686742    4391 cpu_manager.go:414] "containerMap: removing container" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy-web"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686747    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy-web"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686753    4391 cpu_manager.go:414] "containerMap: removing container" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686757    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686768    4391 cpu_manager.go:414] "containerMap: removing container" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy-metric"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686772    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy-metric"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686829    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="prom-label-proxy"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686837    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="alertmanager"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686845    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy-web"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686850    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686855    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="kube-rbac-proxy-metric"
Oct 30 18:38:10.686892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.686861    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="680e4ff2-b452-472a-9f27-96515de0a671" containerName="config-reloader"
Oct 30 18:38:10.690768 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.690754    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.741301 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.741271    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 18:38:10.741747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.741357    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 18:38:10.741747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.741375    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 18:38:10.741747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.741401    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 18:38:10.741747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.741458    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 18:38:10.741747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.741470    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jgbc8\""
Oct 30 18:38:10.741747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.741488    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 18:38:10.741747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.741490    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 18:38:10.741747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.741691    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 18:38:10.763077 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763046    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzfj\" (UniqueName: \"kubernetes.io/projected/4e1f4f57-65ab-421c-a873-ce48186c5115-kube-api-access-zbzfj\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763082    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763110    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e1f4f57-65ab-421c-a873-ce48186c5115-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763166    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763375 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763312    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-web-config\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763375 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763358    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763473 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763444    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e1f4f57-65ab-421c-a873-ce48186c5115-config-out\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763525 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763474    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e1f4f57-65ab-421c-a873-ce48186c5115-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763525 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763500    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e1f4f57-65ab-421c-a873-ce48186c5115-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763624 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763612    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763657 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763643    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763722 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763708    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.763754 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.763730    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1f4f57-65ab-421c-a873-ce48186c5115-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.787769 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.787747    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 18:38:10.865168 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865123    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865355 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865177    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e1f4f57-65ab-421c-a873-ce48186c5115-config-out\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865355 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865204    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e1f4f57-65ab-421c-a873-ce48186c5115-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865355 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865225    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e1f4f57-65ab-421c-a873-ce48186c5115-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865355 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865257    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865355 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865287    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865645 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865422    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865645 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865469    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1f4f57-65ab-421c-a873-ce48186c5115-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865645 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865494    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzfj\" (UniqueName: \"kubernetes.io/projected/4e1f4f57-65ab-421c-a873-ce48186c5115-kube-api-access-zbzfj\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865645 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865520    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865645 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865545    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e1f4f57-65ab-421c-a873-ce48186c5115-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865645 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865603    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865935 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865657    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-web-config\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.865935 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.865724    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e1f4f57-65ab-421c-a873-ce48186c5115-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.866612 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.866251    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e1f4f57-65ab-421c-a873-ce48186c5115-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.866794 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.866769    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1f4f57-65ab-421c-a873-ce48186c5115-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.868411 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.868339    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e1f4f57-65ab-421c-a873-ce48186c5115-config-out\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.868565 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.868534    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.868781 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.868752    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.868874 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.868855    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.868956 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.868940    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.869209 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.869185    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-web-config\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.869279 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.869205    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.869624 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.869607    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e1f4f57-65ab-421c-a873-ce48186c5115-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.870045 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.870012    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e1f4f57-65ab-421c-a873-ce48186c5115-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:10.909913 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.909884    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:10.921544 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:10.921505    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzfj\" (UniqueName: \"kubernetes.io/projected/4e1f4f57-65ab-421c-a873-ce48186c5115-kube-api-access-zbzfj\") pod \"alertmanager-main-0\" (UID: \"4e1f4f57-65ab-421c-a873-ce48186c5115\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:11.004031 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:11.003948    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680e4ff2-b452-472a-9f27-96515de0a671" path="/var/lib/kubelet/pods/680e4ff2-b452-472a-9f27-96515de0a671/volumes"
Oct 30 18:38:11.010163 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:11.010131    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 18:38:11.251497 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:11.251454    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:38:11.429296 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:11.429267    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 18:38:11.432369 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:38:11.432342    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1f4f57_65ab_421c_a873_ce48186c5115.slice/crio-3f9aaf53efd2d5fe8e43143a2eb73aa322adf4603019e5ee4d6e07f7eebc0f94 WatchSource:0}: Error finding container 3f9aaf53efd2d5fe8e43143a2eb73aa322adf4603019e5ee4d6e07f7eebc0f94: Status 404 returned error can't find the container with id 3f9aaf53efd2d5fe8e43143a2eb73aa322adf4603019e5ee4d6e07f7eebc0f94
Oct 30 18:38:11.438320 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:11.438289    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e1f4f57-65ab-421c-a873-ce48186c5115","Type":"ContainerStarted","Data":"3f9aaf53efd2d5fe8e43143a2eb73aa322adf4603019e5ee4d6e07f7eebc0f94"}
Oct 30 18:38:12.028572 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:12.028534    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:12.028981 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:12.028605    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:12.033298 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:12.033277    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:12.443175 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:12.443143    4391 generic.go:356] "Generic (PLEG): container finished" podID="4e1f4f57-65ab-421c-a873-ce48186c5115" containerID="6ca70b4c4cc5226a16e29b9117aa6d3025a4ae34c73c676b5d80e2c786b69a9f" exitCode=0
Oct 30 18:38:12.443320 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:12.443234    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e1f4f57-65ab-421c-a873-ce48186c5115","Type":"ContainerDied","Data":"6ca70b4c4cc5226a16e29b9117aa6d3025a4ae34c73c676b5d80e2c786b69a9f"}
Oct 30 18:38:12.447304 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:12.447278    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57968cdc5f-b5vgr"
Oct 30 18:38:12.785827 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:12.785802    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8547bbb8db-lfnll"]
Oct 30 18:38:13.450967 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:13.450928    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e1f4f57-65ab-421c-a873-ce48186c5115","Type":"ContainerStarted","Data":"59e97def37aec57a97344012e15e4440492c79a30a97e23d4f28e40444c9cd3a"}
Oct 30 18:38:13.451340 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:13.450996    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e1f4f57-65ab-421c-a873-ce48186c5115","Type":"ContainerStarted","Data":"be90e2ab1fafa9daf70335834d3a2dd8b61eae68b27c267ac8ebe961c4036a75"}
Oct 30 18:38:13.451340 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:13.451032    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e1f4f57-65ab-421c-a873-ce48186c5115","Type":"ContainerStarted","Data":"4cb15913a999a84336d97b50364a8784bd722ae3fd9d0fc6bdd7c5cacd15e7c2"}
Oct 30 18:38:13.451340 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:13.451059    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e1f4f57-65ab-421c-a873-ce48186c5115","Type":"ContainerStarted","Data":"af8c9bf996dad87e49e370917f96fe7bd469ea0b1ab441e5f2eb47ca33cc3531"}
Oct 30 18:38:13.451340 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:13.451083    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e1f4f57-65ab-421c-a873-ce48186c5115","Type":"ContainerStarted","Data":"2ebc188295880c32930acc362b0477739ce42e37f336dbdb230e9893189a66b5"}
Oct 30 18:38:13.451340 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:13.451105    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e1f4f57-65ab-421c-a873-ce48186c5115","Type":"ContainerStarted","Data":"d1218719b5fdf105ab86806e61711c421f08e5dfe7e196becdb96d24f6a244da"}
Oct 30 18:38:13.570428 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:13.570372    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.557241437 podStartE2EDuration="3.570357948s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.116511ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:13.568869977 +0000 UTC m=+88.679603273" watchObservedRunningTime="2026-10-30 18:38:13.570357948 +0000 UTC m=+88.681091230"
Oct 30 18:38:37.804395 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:37.804350    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8547bbb8db-lfnll" podUID="7fb969ca-4b98-4926-95ea-6190bd56c62e" containerName="console" containerID="cri-o://bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17" gracePeriod=15
Oct 30 18:38:38.032818 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.032795    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:38:38.098258 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.098183    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-trusted-ca-bundle\") pod \"7fb969ca-4b98-4926-95ea-6190bd56c62e\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") "
Oct 30 18:38:38.098258 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.098221    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb969ca-4b98-4926-95ea-6190bd56c62e-kube-api-access-f4vpd\" (UniqueName: \"kubernetes.io/projected/7fb969ca-4b98-4926-95ea-6190bd56c62e-kube-api-access-f4vpd\") pod \"7fb969ca-4b98-4926-95ea-6190bd56c62e\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") "
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.098661    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-oauth-config\") pod \"7fb969ca-4b98-4926-95ea-6190bd56c62e\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") "
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.098721    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-service-ca\") pod \"7fb969ca-4b98-4926-95ea-6190bd56c62e\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") "
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.098793    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-oauth-serving-cert\") pod \"7fb969ca-4b98-4926-95ea-6190bd56c62e\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") "
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.098901    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-config\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-config\") pod \"7fb969ca-4b98-4926-95ea-6190bd56c62e\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") "
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.098963    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb969ca-4b98-4926-95ea-6190bd56c62e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb969ca-4b98-4926-95ea-6190bd56c62e-tmp\") pod \"7fb969ca-4b98-4926-95ea-6190bd56c62e\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") "
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.099036    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-serving-cert\") pod \"7fb969ca-4b98-4926-95ea-6190bd56c62e\" (UID: \"7fb969ca-4b98-4926-95ea-6190bd56c62e\") "
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.099111    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-trusted-ca-bundle" pod "7fb969ca-4b98-4926-95ea-6190bd56c62e" (UID: "7fb969ca-4b98-4926-95ea-6190bd56c62e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.099156    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-config" pod "7fb969ca-4b98-4926-95ea-6190bd56c62e" (UID: "7fb969ca-4b98-4926-95ea-6190bd56c62e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.099464    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-service-ca" pod "7fb969ca-4b98-4926-95ea-6190bd56c62e" (UID: "7fb969ca-4b98-4926-95ea-6190bd56c62e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.099534    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-oauth-serving-cert" pod "7fb969ca-4b98-4926-95ea-6190bd56c62e" (UID: "7fb969ca-4b98-4926-95ea-6190bd56c62e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.099620    4391 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-config\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.099644    4391 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-trusted-ca-bundle\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:38.100860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.099762    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb969ca-4b98-4926-95ea-6190bd56c62e-tmp" pod "7fb969ca-4b98-4926-95ea-6190bd56c62e" (UID: "7fb969ca-4b98-4926-95ea-6190bd56c62e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:38:38.104518 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.101665    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-oauth-config" pod "7fb969ca-4b98-4926-95ea-6190bd56c62e" (UID: "7fb969ca-4b98-4926-95ea-6190bd56c62e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:38.104968 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.104940    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb969ca-4b98-4926-95ea-6190bd56c62e-kube-api-access-f4vpd" pod "7fb969ca-4b98-4926-95ea-6190bd56c62e" (UID: "7fb969ca-4b98-4926-95ea-6190bd56c62e"). InnerVolumeSpecName "kube-api-access-f4vpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:38.104968 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.104952    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-serving-cert" pod "7fb969ca-4b98-4926-95ea-6190bd56c62e" (UID: "7fb969ca-4b98-4926-95ea-6190bd56c62e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:38.200934 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.200900    4391 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-serving-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:38.200934 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.200928    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4vpd\" (UniqueName: \"kubernetes.io/projected/7fb969ca-4b98-4926-95ea-6190bd56c62e-kube-api-access-f4vpd\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:38.200934 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.200940    4391 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fb969ca-4b98-4926-95ea-6190bd56c62e-console-oauth-config\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:38.201171 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.200951    4391 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:38.201171 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.200960    4391 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fb969ca-4b98-4926-95ea-6190bd56c62e-oauth-serving-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:38.201171 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.200969    4391 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb969ca-4b98-4926-95ea-6190bd56c62e-tmp\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:38.552172 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.552134    4391 generic.go:356] "Generic (PLEG): container finished" podID="7fb969ca-4b98-4926-95ea-6190bd56c62e" containerID="bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17" exitCode=0
Oct 30 18:38:38.552398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.552208    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8547bbb8db-lfnll"
Oct 30 18:38:38.552398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.552220    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8547bbb8db-lfnll" event={"ID":"7fb969ca-4b98-4926-95ea-6190bd56c62e","Type":"ContainerDied","Data":"bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17"}
Oct 30 18:38:38.552398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.552295    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8547bbb8db-lfnll" event={"ID":"7fb969ca-4b98-4926-95ea-6190bd56c62e","Type":"ContainerDied","Data":"6396df837d1928cc78545e24a1a3bc91ba5eddbc416003df5c6c982b5824e828"}
Oct 30 18:38:38.552398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.552316    4391 scope.go:122] "RemoveContainer" containerID="bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17"
Oct 30 18:38:38.560820 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.560804    4391 scope.go:122] "RemoveContainer" containerID="bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17"
Oct 30 18:38:38.561083 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:38.561064    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17\": container with ID starting with bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17 not found: ID does not exist" containerID="bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17"
Oct 30 18:38:38.561129 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.561091    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17"} err="failed to get container status \"bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17\": rpc error: code = NotFound desc = could not find container \"bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17\": container with ID starting with bed29678424a5728b76e8541b510e96f9a4308402eaced189f7e76a78d1a7f17 not found: ID does not exist"
Oct 30 18:38:38.710407 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.710373    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8547bbb8db-lfnll"]
Oct 30 18:38:38.756699 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:38.756665    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8547bbb8db-lfnll"]
Oct 30 18:38:39.003388 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:38:39.003354    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb969ca-4b98-4926-95ea-6190bd56c62e" path="/var/lib/kubelet/pods/7fb969ca-4b98-4926-95ea-6190bd56c62e/volumes"
Oct 30 18:38:41.251353 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:38:41.251320    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:39:11.251713 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:39:11.251632    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:39:41.250550 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:39:41.250512    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:40:02.002799 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.002757    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"]
Oct 30 18:40:02.003442 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.003211    4391 cpu_manager.go:414] "containerMap: removing container" podUID="7fb969ca-4b98-4926-95ea-6190bd56c62e" containerName="console"
Oct 30 18:40:02.003442 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.003227    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb969ca-4b98-4926-95ea-6190bd56c62e" containerName="console"
Oct 30 18:40:02.003442 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.003340    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb969ca-4b98-4926-95ea-6190bd56c62e" containerName="console"
Oct 30 18:40:02.006141 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.006117    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.056602 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.056547    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:40:02.100096 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.100056    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.100297 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.100136    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.100297 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.100236    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8dz5\" (UniqueName: \"kubernetes.io/projected/79f3b664-74c5-4711-8fe9-4f039a5d390f-kube-api-access-d8dz5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.103525 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.103495    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:40:02.103696 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.103509    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-cmrcx\""
Oct 30 18:40:02.106031 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.106002    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"]
Oct 30 18:40:02.201526 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.201482    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.201526 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.201527    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8dz5\" (UniqueName: \"kubernetes.io/projected/79f3b664-74c5-4711-8fe9-4f039a5d390f-kube-api-access-d8dz5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.201742 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.201563    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.201888 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.201868    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.201927 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.201884    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.256687 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.256610    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8dz5\" (UniqueName: \"kubernetes.io/projected/79f3b664-74c5-4711-8fe9-4f039a5d390f-kube-api-access-d8dz5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.320564 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.320521    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:02.683569 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.683537    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"]
Oct 30 18:40:02.687106 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:40:02.687073    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f3b664_74c5_4711_8fe9_4f039a5d390f.slice/crio-5b7218413502e418126cec9cd7a5cb334822a80d86379526862431ea342c4d22 WatchSource:0}: Error finding container 5b7218413502e418126cec9cd7a5cb334822a80d86379526862431ea342c4d22: Status 404 returned error can't find the container with id 5b7218413502e418126cec9cd7a5cb334822a80d86379526862431ea342c4d22
Oct 30 18:40:02.856012 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:02.855975    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8" event={"ID":"79f3b664-74c5-4711-8fe9-4f039a5d390f","Type":"ContainerStarted","Data":"5b7218413502e418126cec9cd7a5cb334822a80d86379526862431ea342c4d22"}
Oct 30 18:40:08.881731 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:08.881695    4391 generic.go:356] "Generic (PLEG): container finished" podID="79f3b664-74c5-4711-8fe9-4f039a5d390f" containerID="135c58ea137a15e1c1f07f7fb7ec2b4adf3d033235970bd35a9bfd319b4ae919" exitCode=0
Oct 30 18:40:08.882208 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:08.881755    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8" event={"ID":"79f3b664-74c5-4711-8fe9-4f039a5d390f","Type":"ContainerDied","Data":"135c58ea137a15e1c1f07f7fb7ec2b4adf3d033235970bd35a9bfd319b4ae919"}
Oct 30 18:40:11.254701 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:40:11.254657    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:40:12.898091 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:12.898059    4391 generic.go:356] "Generic (PLEG): container finished" podID="79f3b664-74c5-4711-8fe9-4f039a5d390f" containerID="ccdca3a98a214c7b34bb22ac1f180b1d59487b7736988708ed483118a2bf04bd" exitCode=0
Oct 30 18:40:12.898533 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:12.898144    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8" event={"ID":"79f3b664-74c5-4711-8fe9-4f039a5d390f","Type":"ContainerDied","Data":"ccdca3a98a214c7b34bb22ac1f180b1d59487b7736988708ed483118a2bf04bd"}
Oct 30 18:40:19.929489 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:19.929454    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8" event={"ID":"79f3b664-74c5-4711-8fe9-4f039a5d390f","Type":"ContainerStarted","Data":"32d7e9beaf326f9b175f2b1ccfbb0c0822cc05663786846f8c4724599515a59d"}
Oct 30 18:40:20.041768 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:20.041715    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8" podStartSLOduration=3.267975158 podStartE2EDuration="19.041701186s" totalImagesPullingTime="15.773726028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:20.040902894 +0000 UTC m=+215.151636173" watchObservedRunningTime="2026-10-30 18:40:20.041701186 +0000 UTC m=+215.152434469"
Oct 30 18:40:20.940639 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:20.940602    4391 generic.go:356] "Generic (PLEG): container finished" podID="79f3b664-74c5-4711-8fe9-4f039a5d390f" containerID="32d7e9beaf326f9b175f2b1ccfbb0c0822cc05663786846f8c4724599515a59d" exitCode=0
Oct 30 18:40:20.941020 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:20.940663    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8" event={"ID":"79f3b664-74c5-4711-8fe9-4f039a5d390f","Type":"ContainerDied","Data":"32d7e9beaf326f9b175f2b1ccfbb0c0822cc05663786846f8c4724599515a59d"}
Oct 30 18:40:22.064514 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.064490    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:22.173830 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.173795    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-bundle\") pod \"79f3b664-74c5-4711-8fe9-4f039a5d390f\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") "
Oct 30 18:40:22.173996 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.173854    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f3b664-74c5-4711-8fe9-4f039a5d390f-kube-api-access-d8dz5\" (UniqueName: \"kubernetes.io/projected/79f3b664-74c5-4711-8fe9-4f039a5d390f-kube-api-access-d8dz5\") pod \"79f3b664-74c5-4711-8fe9-4f039a5d390f\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") "
Oct 30 18:40:22.173996 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.173928    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-util\" (UniqueName: \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-util\") pod \"79f3b664-74c5-4711-8fe9-4f039a5d390f\" (UID: \"79f3b664-74c5-4711-8fe9-4f039a5d390f\") "
Oct 30 18:40:22.174472 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.174444    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-bundle" pod "79f3b664-74c5-4711-8fe9-4f039a5d390f" (UID: "79f3b664-74c5-4711-8fe9-4f039a5d390f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:40:22.176376 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.176350    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f3b664-74c5-4711-8fe9-4f039a5d390f-kube-api-access-d8dz5" pod "79f3b664-74c5-4711-8fe9-4f039a5d390f" (UID: "79f3b664-74c5-4711-8fe9-4f039a5d390f"). InnerVolumeSpecName "kube-api-access-d8dz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:40:22.180601 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.180545    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-util" pod "79f3b664-74c5-4711-8fe9-4f039a5d390f" (UID: "79f3b664-74c5-4711-8fe9-4f039a5d390f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:40:22.274646 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.274539    4391 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-bundle\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:22.274646 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.274600    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8dz5\" (UniqueName: \"kubernetes.io/projected/79f3b664-74c5-4711-8fe9-4f039a5d390f-kube-api-access-d8dz5\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:22.274646 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.274616    4391 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79f3b664-74c5-4711-8fe9-4f039a5d390f-util\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:22.949932 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.949905    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8"
Oct 30 18:40:22.949932 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.949927    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs96d8" event={"ID":"79f3b664-74c5-4711-8fe9-4f039a5d390f","Type":"ContainerDied","Data":"5b7218413502e418126cec9cd7a5cb334822a80d86379526862431ea342c4d22"}
Oct 30 18:40:22.950139 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:22.949973    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b7218413502e418126cec9cd7a5cb334822a80d86379526862431ea342c4d22"
Oct 30 18:40:29.848867 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.848780    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"]
Oct 30 18:40:29.849324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.849131    4391 cpu_manager.go:414] "containerMap: removing container" podUID="79f3b664-74c5-4711-8fe9-4f039a5d390f" containerName="extract"
Oct 30 18:40:29.849324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.849143    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f3b664-74c5-4711-8fe9-4f039a5d390f" containerName="extract"
Oct 30 18:40:29.849324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.849162    4391 cpu_manager.go:414] "containerMap: removing container" podUID="79f3b664-74c5-4711-8fe9-4f039a5d390f" containerName="util"
Oct 30 18:40:29.849324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.849166    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f3b664-74c5-4711-8fe9-4f039a5d390f" containerName="util"
Oct 30 18:40:29.849324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.849177    4391 cpu_manager.go:414] "containerMap: removing container" podUID="79f3b664-74c5-4711-8fe9-4f039a5d390f" containerName="pull"
Oct 30 18:40:29.849324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.849183    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f3b664-74c5-4711-8fe9-4f039a5d390f" containerName="pull"
Oct 30 18:40:29.849324 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.849236    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f3b664-74c5-4711-8fe9-4f039a5d390f" containerName="extract"
Oct 30 18:40:29.851495 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.851476    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"
Oct 30 18:40:29.898484 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.898452    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"]
Oct 30 18:40:29.938493 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.938457    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6022d7a9-7fc2-4995-ae17-ed0cd7b91993-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-pgfcd\" (UID: \"6022d7a9-7fc2-4995-ae17-ed0cd7b91993\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"
Oct 30 18:40:29.938664 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.938515    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l25bf\" (UniqueName: \"kubernetes.io/projected/6022d7a9-7fc2-4995-ae17-ed0cd7b91993-kube-api-access-l25bf\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-pgfcd\" (UID: \"6022d7a9-7fc2-4995-ae17-ed0cd7b91993\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"
Oct 30 18:40:29.943771 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.943749    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:40:29.943875 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.943793    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 18:40:29.943931 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:29.943895    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rd8zr\""
Oct 30 18:40:30.039661 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:30.039620    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6022d7a9-7fc2-4995-ae17-ed0cd7b91993-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-pgfcd\" (UID: \"6022d7a9-7fc2-4995-ae17-ed0cd7b91993\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"
Oct 30 18:40:30.039845 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:30.039742    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25bf\" (UniqueName: \"kubernetes.io/projected/6022d7a9-7fc2-4995-ae17-ed0cd7b91993-kube-api-access-l25bf\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-pgfcd\" (UID: \"6022d7a9-7fc2-4995-ae17-ed0cd7b91993\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"
Oct 30 18:40:30.040043 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:30.040022    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6022d7a9-7fc2-4995-ae17-ed0cd7b91993-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-pgfcd\" (UID: \"6022d7a9-7fc2-4995-ae17-ed0cd7b91993\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"
Oct 30 18:40:30.095662 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:30.095637    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25bf\" (UniqueName: \"kubernetes.io/projected/6022d7a9-7fc2-4995-ae17-ed0cd7b91993-kube-api-access-l25bf\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-pgfcd\" (UID: \"6022d7a9-7fc2-4995-ae17-ed0cd7b91993\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"
Oct 30 18:40:30.164350 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:30.164303    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"
Oct 30 18:40:30.584616 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:30.584510    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd"]
Oct 30 18:40:30.590269 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:40:30.590239    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6022d7a9_7fc2_4995_ae17_ed0cd7b91993.slice/crio-78bfcbb4145b51a1c18e02025973e71838f7c2dbf87966adfdf8249b09b47984 WatchSource:0}: Error finding container 78bfcbb4145b51a1c18e02025973e71838f7c2dbf87966adfdf8249b09b47984: Status 404 returned error can't find the container with id 78bfcbb4145b51a1c18e02025973e71838f7c2dbf87966adfdf8249b09b47984
Oct 30 18:40:30.979453 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:30.979422    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd" event={"ID":"6022d7a9-7fc2-4995-ae17-ed0cd7b91993","Type":"ContainerStarted","Data":"78bfcbb4145b51a1c18e02025973e71838f7c2dbf87966adfdf8249b09b47984"}
Oct 30 18:40:34.998290 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:34.998249    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd" event={"ID":"6022d7a9-7fc2-4995-ae17-ed0cd7b91993","Type":"ContainerStarted","Data":"699d4296ca5b5fa51909f9f7f1ae1247cb40f4ab196f7e1e3c10ea26bdf68bc8"}
Oct 30 18:40:35.205257 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:35.205205    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-pgfcd" podStartSLOduration=2.8545159829999998 podStartE2EDuration="6.20519377s" totalImagesPullingTime="3.350677787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:35.158407349 +0000 UTC m=+230.269140631" watchObservedRunningTime="2026-10-30 18:40:35.20519377 +0000 UTC m=+230.315927052"
Oct 30 18:40:37.639990 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.639955    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"]
Oct 30 18:40:37.642668 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.642646    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"
Oct 30 18:40:37.704282 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.704252    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0d13956-1aa6-413b-8e55-0b4d755b5084-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fw8g8\" (UID: \"e0d13956-1aa6-413b-8e55-0b4d755b5084\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"
Oct 30 18:40:37.704437 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.704305    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg5z5\" (UniqueName: \"kubernetes.io/projected/e0d13956-1aa6-413b-8e55-0b4d755b5084-kube-api-access-tg5z5\") pod \"cert-manager-cainjector-5c57bbd4fc-fw8g8\" (UID: \"e0d13956-1aa6-413b-8e55-0b4d755b5084\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"
Oct 30 18:40:37.706672 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.706648    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"]
Oct 30 18:40:37.739745 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.739722    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 18:40:37.740064 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.740048    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-62nbh\""
Oct 30 18:40:37.740125 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.740085    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 18:40:37.753154 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.753129    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"]
Oct 30 18:40:37.755753 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.755735    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:37.804919 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.804889    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:37.805078 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.804935    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0d13956-1aa6-413b-8e55-0b4d755b5084-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fw8g8\" (UID: \"e0d13956-1aa6-413b-8e55-0b4d755b5084\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"
Oct 30 18:40:37.805078 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.804957    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:37.805161 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.805095    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg5z5\" (UniqueName: \"kubernetes.io/projected/e0d13956-1aa6-413b-8e55-0b4d755b5084-kube-api-access-tg5z5\") pod \"cert-manager-cainjector-5c57bbd4fc-fw8g8\" (UID: \"e0d13956-1aa6-413b-8e55-0b4d755b5084\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"
Oct 30 18:40:37.805161 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.805143    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9v9n\" (UniqueName: \"kubernetes.io/projected/63149a29-dc6f-4903-be24-cf8080adb7e0-kube-api-access-l9v9n\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:37.805831 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.805810    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:40:37.805987 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.805972    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:40:37.806244 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.806227    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-qsf6m\""
Oct 30 18:40:37.810925 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.810905    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"]
Oct 30 18:40:37.860003 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.859975    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0d13956-1aa6-413b-8e55-0b4d755b5084-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fw8g8\" (UID: \"e0d13956-1aa6-413b-8e55-0b4d755b5084\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"
Oct 30 18:40:37.860125 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.859974    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg5z5\" (UniqueName: \"kubernetes.io/projected/e0d13956-1aa6-413b-8e55-0b4d755b5084-kube-api-access-tg5z5\") pod \"cert-manager-cainjector-5c57bbd4fc-fw8g8\" (UID: \"e0d13956-1aa6-413b-8e55-0b4d755b5084\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"
Oct 30 18:40:37.905871 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.905841    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9v9n\" (UniqueName: \"kubernetes.io/projected/63149a29-dc6f-4903-be24-cf8080adb7e0-kube-api-access-l9v9n\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:37.906023 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.905901    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:37.906023 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.905942    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:37.906296 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.906278    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:37.906341 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.906313    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:37.960219 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.960188    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9v9n\" (UniqueName: \"kubernetes.io/projected/63149a29-dc6f-4903-be24-cf8080adb7e0-kube-api-access-l9v9n\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:37.968598 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:37.968561    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"
Oct 30 18:40:38.073663 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:38.073635    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:38.334251 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:38.334177    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8"]
Oct 30 18:40:38.337388 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:40:38.337357    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0d13956_1aa6_413b_8e55_0b4d755b5084.slice/crio-3d9c9c830f965c89be39fe3b6a7886e956b857f4c3ade2ce8dd1c5f6b3fc933d WatchSource:0}: Error finding container 3d9c9c830f965c89be39fe3b6a7886e956b857f4c3ade2ce8dd1c5f6b3fc933d: Status 404 returned error can't find the container with id 3d9c9c830f965c89be39fe3b6a7886e956b857f4c3ade2ce8dd1c5f6b3fc933d
Oct 30 18:40:38.442097 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:38.441879    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"]
Oct 30 18:40:38.446972 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:40:38.446947    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63149a29_dc6f_4903_be24_cf8080adb7e0.slice/crio-1ce971a715b9ac62981c0bb9d95a0db8659e75af506f9d4285a610c0723897fc WatchSource:0}: Error finding container 1ce971a715b9ac62981c0bb9d95a0db8659e75af506f9d4285a610c0723897fc: Status 404 returned error can't find the container with id 1ce971a715b9ac62981c0bb9d95a0db8659e75af506f9d4285a610c0723897fc
Oct 30 18:40:39.015798 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:39.015759    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8" event={"ID":"e0d13956-1aa6-413b-8e55-0b4d755b5084","Type":"ContainerStarted","Data":"3d9c9c830f965c89be39fe3b6a7886e956b857f4c3ade2ce8dd1c5f6b3fc933d"}
Oct 30 18:40:39.017181 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:39.017158    4391 generic.go:356] "Generic (PLEG): container finished" podID="63149a29-dc6f-4903-be24-cf8080adb7e0" containerID="c37ca048e7e5789d3dbb0aa44743b0234c64276dd9e303ae771e611534e6e8d8" exitCode=0
Oct 30 18:40:39.017263 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:39.017198    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd" event={"ID":"63149a29-dc6f-4903-be24-cf8080adb7e0","Type":"ContainerDied","Data":"c37ca048e7e5789d3dbb0aa44743b0234c64276dd9e303ae771e611534e6e8d8"}
Oct 30 18:40:39.017263 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:39.017225    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd" event={"ID":"63149a29-dc6f-4903-be24-cf8080adb7e0","Type":"ContainerStarted","Data":"1ce971a715b9ac62981c0bb9d95a0db8659e75af506f9d4285a610c0723897fc"}
Oct 30 18:40:41.260918 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:40:41.260848    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:40:43.036646 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:43.036613    4391 generic.go:356] "Generic (PLEG): container finished" podID="63149a29-dc6f-4903-be24-cf8080adb7e0" containerID="51e93443c04e5273f4e956b2feab8a6aaa34605b2827b610c059c4b284dfc083" exitCode=0
Oct 30 18:40:43.037098 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:43.036697    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd" event={"ID":"63149a29-dc6f-4903-be24-cf8080adb7e0","Type":"ContainerDied","Data":"51e93443c04e5273f4e956b2feab8a6aaa34605b2827b610c059c4b284dfc083"}
Oct 30 18:40:43.038430 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:43.038408    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8" event={"ID":"e0d13956-1aa6-413b-8e55-0b4d755b5084","Type":"ContainerStarted","Data":"b8303263b6aeaf585cbbca3bc1f6857f06081409eb1f5789cd321a6ba4df7713"}
Oct 30 18:40:43.255599 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:43.255535    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fw8g8" podStartSLOduration=2.496624776 podStartE2EDuration="6.255521777s" totalImagesPullingTime="3.758897001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:43.254997199 +0000 UTC m=+238.365730485" watchObservedRunningTime="2026-10-30 18:40:43.255521777 +0000 UTC m=+238.366255059"
Oct 30 18:40:44.047789 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:44.047753    4391 generic.go:356] "Generic (PLEG): container finished" podID="63149a29-dc6f-4903-be24-cf8080adb7e0" containerID="9fe969393982d6946de74f39788ba80078d7e18209f39fca278a396a32960e9a" exitCode=0
Oct 30 18:40:44.048183 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:44.047834    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd" event={"ID":"63149a29-dc6f-4903-be24-cf8080adb7e0","Type":"ContainerDied","Data":"9fe969393982d6946de74f39788ba80078d7e18209f39fca278a396a32960e9a"}
Oct 30 18:40:45.175688 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:45.175664    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:45.269038 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:45.269006    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-bundle\") pod \"63149a29-dc6f-4903-be24-cf8080adb7e0\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") "
Oct 30 18:40:45.269203 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:45.269069    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63149a29-dc6f-4903-be24-cf8080adb7e0-kube-api-access-l9v9n\" (UniqueName: \"kubernetes.io/projected/63149a29-dc6f-4903-be24-cf8080adb7e0-kube-api-access-l9v9n\") pod \"63149a29-dc6f-4903-be24-cf8080adb7e0\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") "
Oct 30 18:40:45.269203 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:45.269142    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-util\" (UniqueName: \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-util\") pod \"63149a29-dc6f-4903-be24-cf8080adb7e0\" (UID: \"63149a29-dc6f-4903-be24-cf8080adb7e0\") "
Oct 30 18:40:45.269438 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:45.269408    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-bundle" pod "63149a29-dc6f-4903-be24-cf8080adb7e0" (UID: "63149a29-dc6f-4903-be24-cf8080adb7e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:40:45.271168 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:45.271143    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63149a29-dc6f-4903-be24-cf8080adb7e0-kube-api-access-l9v9n" pod "63149a29-dc6f-4903-be24-cf8080adb7e0" (UID: "63149a29-dc6f-4903-be24-cf8080adb7e0"). InnerVolumeSpecName "kube-api-access-l9v9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:40:45.273595 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:45.273561    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-util" pod "63149a29-dc6f-4903-be24-cf8080adb7e0" (UID: "63149a29-dc6f-4903-be24-cf8080adb7e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:40:45.370404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:45.370326    4391 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-bundle\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:45.370404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:45.370354    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9v9n\" (UniqueName: \"kubernetes.io/projected/63149a29-dc6f-4903-be24-cf8080adb7e0-kube-api-access-l9v9n\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:45.370404 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:45.370366    4391 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63149a29-dc6f-4903-be24-cf8080adb7e0-util\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:46.058099 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:46.058069    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd"
Oct 30 18:40:46.058099 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:46.058079    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85gj4pd" event={"ID":"63149a29-dc6f-4903-be24-cf8080adb7e0","Type":"ContainerDied","Data":"1ce971a715b9ac62981c0bb9d95a0db8659e75af506f9d4285a610c0723897fc"}
Oct 30 18:40:46.058302 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:46.058121    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ce971a715b9ac62981c0bb9d95a0db8659e75af506f9d4285a610c0723897fc"
Oct 30 18:40:56.232883 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.232848    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-h5775"]
Oct 30 18:40:56.233353 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.233177    4391 cpu_manager.go:414] "containerMap: removing container" podUID="63149a29-dc6f-4903-be24-cf8080adb7e0" containerName="extract"
Oct 30 18:40:56.233353 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.233188    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63149a29-dc6f-4903-be24-cf8080adb7e0" containerName="extract"
Oct 30 18:40:56.233353 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.233208    4391 cpu_manager.go:414] "containerMap: removing container" podUID="63149a29-dc6f-4903-be24-cf8080adb7e0" containerName="util"
Oct 30 18:40:56.233353 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.233214    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63149a29-dc6f-4903-be24-cf8080adb7e0" containerName="util"
Oct 30 18:40:56.233353 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.233222    4391 cpu_manager.go:414] "containerMap: removing container" podUID="63149a29-dc6f-4903-be24-cf8080adb7e0" containerName="pull"
Oct 30 18:40:56.233353 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.233227    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63149a29-dc6f-4903-be24-cf8080adb7e0" containerName="pull"
Oct 30 18:40:56.233353 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.233288    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="63149a29-dc6f-4903-be24-cf8080adb7e0" containerName="extract"
Oct 30 18:40:56.235761 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.235741    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-h5775"
Oct 30 18:40:56.331760 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.331723    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5kdt4\""
Oct 30 18:40:56.333016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.332993    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-h5775"]
Oct 30 18:40:56.363912 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.363873    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45rq\" (UniqueName: \"kubernetes.io/projected/c9b2f852-70cc-446e-beae-70af33424531-kube-api-access-v45rq\") pod \"cert-manager-597b95cb84-h5775\" (UID: \"c9b2f852-70cc-446e-beae-70af33424531\") " pod="cert-manager/cert-manager-597b95cb84-h5775"
Oct 30 18:40:56.364083 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.363934    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9b2f852-70cc-446e-beae-70af33424531-bound-sa-token\") pod \"cert-manager-597b95cb84-h5775\" (UID: \"c9b2f852-70cc-446e-beae-70af33424531\") " pod="cert-manager/cert-manager-597b95cb84-h5775"
Oct 30 18:40:56.465186 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.465151    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9b2f852-70cc-446e-beae-70af33424531-bound-sa-token\") pod \"cert-manager-597b95cb84-h5775\" (UID: \"c9b2f852-70cc-446e-beae-70af33424531\") " pod="cert-manager/cert-manager-597b95cb84-h5775"
Oct 30 18:40:56.465332 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.465216    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45rq\" (UniqueName: \"kubernetes.io/projected/c9b2f852-70cc-446e-beae-70af33424531-kube-api-access-v45rq\") pod \"cert-manager-597b95cb84-h5775\" (UID: \"c9b2f852-70cc-446e-beae-70af33424531\") " pod="cert-manager/cert-manager-597b95cb84-h5775"
Oct 30 18:40:56.521443 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.521386    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9b2f852-70cc-446e-beae-70af33424531-bound-sa-token\") pod \"cert-manager-597b95cb84-h5775\" (UID: \"c9b2f852-70cc-446e-beae-70af33424531\") " pod="cert-manager/cert-manager-597b95cb84-h5775"
Oct 30 18:40:56.521983 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.521964    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45rq\" (UniqueName: \"kubernetes.io/projected/c9b2f852-70cc-446e-beae-70af33424531-kube-api-access-v45rq\") pod \"cert-manager-597b95cb84-h5775\" (UID: \"c9b2f852-70cc-446e-beae-70af33424531\") " pod="cert-manager/cert-manager-597b95cb84-h5775"
Oct 30 18:40:56.546801 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.546774    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-h5775"
Oct 30 18:40:56.907770 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:56.907678    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-h5775"]
Oct 30 18:40:56.910474 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:40:56.910449    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9b2f852_70cc_446e_beae_70af33424531.slice/crio-8babb8b7efee463325829119e3603f15b7785b5327f4ec365a112b08ab085307 WatchSource:0}: Error finding container 8babb8b7efee463325829119e3603f15b7785b5327f4ec365a112b08ab085307: Status 404 returned error can't find the container with id 8babb8b7efee463325829119e3603f15b7785b5327f4ec365a112b08ab085307
Oct 30 18:40:57.107562 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:57.107526    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-h5775" event={"ID":"c9b2f852-70cc-446e-beae-70af33424531","Type":"ContainerStarted","Data":"ceee0eb7ffbd9907a086c0c507913bafbead6a72e06fa26f05ad4a0b6106f608"}
Oct 30 18:40:57.107562 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:57.107566    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-h5775" event={"ID":"c9b2f852-70cc-446e-beae-70af33424531","Type":"ContainerStarted","Data":"8babb8b7efee463325829119e3603f15b7785b5327f4ec365a112b08ab085307"}
Oct 30 18:40:57.224568 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:57.224470    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-h5775" podStartSLOduration=1.224456508 podStartE2EDuration="1.224456508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:57.223598506 +0000 UTC m=+252.334331794" watchObservedRunningTime="2026-10-30 18:40:57.224456508 +0000 UTC m=+252.335189791"
Oct 30 18:40:59.908655 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:59.908614    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"]
Oct 30 18:40:59.911605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:59.911570    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:40:59.995436 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:59.995392    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndchw\" (UniqueName: \"kubernetes.io/projected/c0fb66bc-64b8-43b7-acba-1a196815408d-kube-api-access-ndchw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:40:59.995641 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:59.995453    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:40:59.995641 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:40:59.995543    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:41:00.007704 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.007674    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:41:00.007863 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.007723    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-rbbtl\""
Oct 30 18:41:00.007863 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.007771    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:41:00.009163 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.009128    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"]
Oct 30 18:41:00.096599 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.096558    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:41:00.096971 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.096949    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:41:00.097089 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.096946    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndchw\" (UniqueName: \"kubernetes.io/projected/c0fb66bc-64b8-43b7-acba-1a196815408d-kube-api-access-ndchw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:41:00.097175 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.097156    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:41:00.097460 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.097438    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:41:00.151940 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.151917    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndchw\" (UniqueName: \"kubernetes.io/projected/c0fb66bc-64b8-43b7-acba-1a196815408d-kube-api-access-ndchw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:41:00.226149 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.226078    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:41:00.589759 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:00.589686    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"]
Oct 30 18:41:00.592654 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:41:00.592628    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fb66bc_64b8_43b7_acba_1a196815408d.slice/crio-24266802a101a0d5aeb243bed64179b3d191c48bb42ca6121f114c24cf2a9d41 WatchSource:0}: Error finding container 24266802a101a0d5aeb243bed64179b3d191c48bb42ca6121f114c24cf2a9d41: Status 404 returned error can't find the container with id 24266802a101a0d5aeb243bed64179b3d191c48bb42ca6121f114c24cf2a9d41
Oct 30 18:41:01.125016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:01.124984    4391 generic.go:356] "Generic (PLEG): container finished" podID="c0fb66bc-64b8-43b7-acba-1a196815408d" containerID="28c4369d2a890701517cd6b57e86059afc979a2ae6296603171227d90d80fabd" exitCode=0
Oct 30 18:41:01.125436 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:01.125035    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd" event={"ID":"c0fb66bc-64b8-43b7-acba-1a196815408d","Type":"ContainerDied","Data":"28c4369d2a890701517cd6b57e86059afc979a2ae6296603171227d90d80fabd"}
Oct 30 18:41:01.125436 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:01.125069    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd" event={"ID":"c0fb66bc-64b8-43b7-acba-1a196815408d","Type":"ContainerStarted","Data":"24266802a101a0d5aeb243bed64179b3d191c48bb42ca6121f114c24cf2a9d41"}
Oct 30 18:41:03.134348 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:03.134315    4391 generic.go:356] "Generic (PLEG): container finished" podID="c0fb66bc-64b8-43b7-acba-1a196815408d" containerID="3a033629542e1ebc629ddff04330b7e935294094f213913d99aac7833e3c0021" exitCode=0
Oct 30 18:41:03.134348 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:03.134344    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd" event={"ID":"c0fb66bc-64b8-43b7-acba-1a196815408d","Type":"ContainerDied","Data":"3a033629542e1ebc629ddff04330b7e935294094f213913d99aac7833e3c0021"}
Oct 30 18:41:04.140548 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:04.140469    4391 generic.go:356] "Generic (PLEG): container finished" podID="c0fb66bc-64b8-43b7-acba-1a196815408d" containerID="38207870c261af18a38671cbe13f6d61e25109faed4ec271fe0062c26ef711dd" exitCode=0
Oct 30 18:41:04.140548 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:04.140526    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd" event={"ID":"c0fb66bc-64b8-43b7-acba-1a196815408d","Type":"ContainerDied","Data":"38207870c261af18a38671cbe13f6d61e25109faed4ec271fe0062c26ef711dd"}
Oct 30 18:41:05.267567 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:05.267545    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:41:05.443892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:05.443861    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fb66bc-64b8-43b7-acba-1a196815408d-kube-api-access-ndchw\" (UniqueName: \"kubernetes.io/projected/c0fb66bc-64b8-43b7-acba-1a196815408d-kube-api-access-ndchw\") pod \"c0fb66bc-64b8-43b7-acba-1a196815408d\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") "
Oct 30 18:41:05.444048 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:05.443906    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-bundle\") pod \"c0fb66bc-64b8-43b7-acba-1a196815408d\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") "
Oct 30 18:41:05.444048 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:05.443963    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-util\" (UniqueName: \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-util\") pod \"c0fb66bc-64b8-43b7-acba-1a196815408d\" (UID: \"c0fb66bc-64b8-43b7-acba-1a196815408d\") "
Oct 30 18:41:05.444632 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:05.444605    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-bundle" pod "c0fb66bc-64b8-43b7-acba-1a196815408d" (UID: "c0fb66bc-64b8-43b7-acba-1a196815408d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:41:05.445988 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:05.445963    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fb66bc-64b8-43b7-acba-1a196815408d-kube-api-access-ndchw" pod "c0fb66bc-64b8-43b7-acba-1a196815408d" (UID: "c0fb66bc-64b8-43b7-acba-1a196815408d"). InnerVolumeSpecName "kube-api-access-ndchw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:41:05.449437 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:05.449416    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-util" pod "c0fb66bc-64b8-43b7-acba-1a196815408d" (UID: "c0fb66bc-64b8-43b7-acba-1a196815408d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:41:05.544497 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:05.544462    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndchw\" (UniqueName: \"kubernetes.io/projected/c0fb66bc-64b8-43b7-acba-1a196815408d-kube-api-access-ndchw\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:41:05.544497 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:05.544491    4391 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-bundle\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:41:05.544497 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:05.544501    4391 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0fb66bc-64b8-43b7-acba-1a196815408d-util\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:41:06.150439 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:06.150413    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd"
Oct 30 18:41:06.150619 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:06.150412    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71c8ndd" event={"ID":"c0fb66bc-64b8-43b7-acba-1a196815408d","Type":"ContainerDied","Data":"24266802a101a0d5aeb243bed64179b3d191c48bb42ca6121f114c24cf2a9d41"}
Oct 30 18:41:06.150619 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:06.150529    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24266802a101a0d5aeb243bed64179b3d191c48bb42ca6121f114c24cf2a9d41"
Oct 30 18:41:11.251340 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:41:11.251302    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:41:15.506666 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.506629    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"]
Oct 30 18:41:15.507181 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.507165    4391 cpu_manager.go:414] "containerMap: removing container" podUID="c0fb66bc-64b8-43b7-acba-1a196815408d" containerName="extract"
Oct 30 18:41:15.507259 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.507183    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fb66bc-64b8-43b7-acba-1a196815408d" containerName="extract"
Oct 30 18:41:15.507259 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.507196    4391 cpu_manager.go:414] "containerMap: removing container" podUID="c0fb66bc-64b8-43b7-acba-1a196815408d" containerName="pull"
Oct 30 18:41:15.507259 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.507204    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fb66bc-64b8-43b7-acba-1a196815408d" containerName="pull"
Oct 30 18:41:15.507259 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.507228    4391 cpu_manager.go:414] "containerMap: removing container" podUID="c0fb66bc-64b8-43b7-acba-1a196815408d" containerName="util"
Oct 30 18:41:15.507259 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.507238    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fb66bc-64b8-43b7-acba-1a196815408d" containerName="util"
Oct 30 18:41:15.507478 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.507322    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fb66bc-64b8-43b7-acba-1a196815408d" containerName="extract"
Oct 30 18:41:15.512027 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.512011    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.523519 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.523495    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66e98c62-ea4d-4305-b143-61c2ce132dcc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.523612 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.523556    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdmb7\" (UniqueName: \"kubernetes.io/projected/66e98c62-ea4d-4305-b143-61c2ce132dcc-kube-api-access-vdmb7\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.523680 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.523662    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e98c62-ea4d-4305-b143-61c2ce132dcc-tmp\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.523726 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.523713    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/66e98c62-ea4d-4305-b143-61c2ce132dcc-metrics-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.523764 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.523734    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66e98c62-ea4d-4305-b143-61c2ce132dcc-webhook-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.523797 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.523764    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66e98c62-ea4d-4305-b143-61c2ce132dcc-manifests\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.567937 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.567903    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 18:41:15.608922 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.608889    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 18:41:15.609093 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.608929    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 18:41:15.609093 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.608928    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 18:41:15.609093 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.609031    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t8d96\""
Oct 30 18:41:15.609273 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.609134    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 18:41:15.611288 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.611255    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"]
Oct 30 18:41:15.624662 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.624630    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdmb7\" (UniqueName: \"kubernetes.io/projected/66e98c62-ea4d-4305-b143-61c2ce132dcc-kube-api-access-vdmb7\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.624792 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.624683    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e98c62-ea4d-4305-b143-61c2ce132dcc-tmp\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.624792 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.624723    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/66e98c62-ea4d-4305-b143-61c2ce132dcc-metrics-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.624792 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.624743    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66e98c62-ea4d-4305-b143-61c2ce132dcc-webhook-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.624792 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.624774    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66e98c62-ea4d-4305-b143-61c2ce132dcc-manifests\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.625001 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.624816    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66e98c62-ea4d-4305-b143-61c2ce132dcc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.625222 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.625196    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66e98c62-ea4d-4305-b143-61c2ce132dcc-manifests\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.627422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.627392    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66e98c62-ea4d-4305-b143-61c2ce132dcc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.627513 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.627470    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66e98c62-ea4d-4305-b143-61c2ce132dcc-webhook-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.627649 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.627632    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e98c62-ea4d-4305-b143-61c2ce132dcc-tmp\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.627693 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.627654    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/66e98c62-ea4d-4305-b143-61c2ce132dcc-metrics-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.684720 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.684684    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdmb7\" (UniqueName: \"kubernetes.io/projected/66e98c62-ea4d-4305-b143-61c2ce132dcc-kube-api-access-vdmb7\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s\" (UID: \"66e98c62-ea4d-4305-b143-61c2ce132dcc\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:15.826093 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:15.826017    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:16.299431 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:16.299406    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"]
Oct 30 18:41:16.301599 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:41:16.301557    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e98c62_ea4d_4305_b143_61c2ce132dcc.slice/crio-04eac346526a58ced4016ddbdb6d064b26a2a1f40a8b1a8819513da55a52ce6e WatchSource:0}: Error finding container 04eac346526a58ced4016ddbdb6d064b26a2a1f40a8b1a8819513da55a52ce6e: Status 404 returned error can't find the container with id 04eac346526a58ced4016ddbdb6d064b26a2a1f40a8b1a8819513da55a52ce6e
Oct 30 18:41:17.197979 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:17.197943    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s" event={"ID":"66e98c62-ea4d-4305-b143-61c2ce132dcc","Type":"ContainerStarted","Data":"04eac346526a58ced4016ddbdb6d064b26a2a1f40a8b1a8819513da55a52ce6e"}
Oct 30 18:41:21.215960 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:21.215919    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s" event={"ID":"66e98c62-ea4d-4305-b143-61c2ce132dcc","Type":"ContainerStarted","Data":"189c7b0c045bce270cee09a2088e394054d6774cefa1584b8efcd8367b781626"}
Oct 30 18:41:22.221516 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:22.221484    4391 generic.go:356] "Generic (PLEG): container finished" podID="66e98c62-ea4d-4305-b143-61c2ce132dcc" containerID="189c7b0c045bce270cee09a2088e394054d6774cefa1584b8efcd8367b781626" exitCode=0
Oct 30 18:41:22.221946 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:22.221543    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s" event={"ID":"66e98c62-ea4d-4305-b143-61c2ce132dcc","Type":"ContainerDied","Data":"189c7b0c045bce270cee09a2088e394054d6774cefa1584b8efcd8367b781626"}
Oct 30 18:41:23.232943 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:23.232903    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s" event={"ID":"66e98c62-ea4d-4305-b143-61c2ce132dcc","Type":"ContainerStarted","Data":"6796fc57b9d7deeee865bd25ceddc972ed24409c5e99f986bf8ec6d8046ac5b6"}
Oct 30 18:41:23.233429 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:23.233031    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:23.401752 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:23.401658    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s" podStartSLOduration=2.406552779 podStartE2EDuration="8.401644949s" totalImagesPullingTime="5.570098038s" totalInitContainerRuntime="424.994132ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:23.400483036 +0000 UTC m=+278.511216320" watchObservedRunningTime="2026-10-30 18:41:23.401644949 +0000 UTC m=+278.512378232"
Oct 30 18:41:34.239255 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:34.239224    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s"
Oct 30 18:41:38.946480 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:38.946445    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"]
Oct 30 18:41:38.950225 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:38.950206    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:38.999532 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:38.999503    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 18:41:39.028361 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.028333    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0196e473-3f42-4506-bedd-da006253b9de-tmp\") pod \"kube-auth-proxy-58d577dc79-jtwnc\" (UID: \"0196e473-3f42-4506-bedd-da006253b9de\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:39.028524 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.028368    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bb4\" (UniqueName: \"kubernetes.io/projected/0196e473-3f42-4506-bedd-da006253b9de-kube-api-access-q5bb4\") pod \"kube-auth-proxy-58d577dc79-jtwnc\" (UID: \"0196e473-3f42-4506-bedd-da006253b9de\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:39.028524 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.028405    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0196e473-3f42-4506-bedd-da006253b9de-tls-certs\") pod \"kube-auth-proxy-58d577dc79-jtwnc\" (UID: \"0196e473-3f42-4506-bedd-da006253b9de\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:39.046407 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.046382    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 18:41:39.046407 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.046390    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5dblg\""
Oct 30 18:41:39.046613 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.046475    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:41:39.046613 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.046507    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:41:39.048128 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.048104    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"]
Oct 30 18:41:39.128985 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.128953    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0196e473-3f42-4506-bedd-da006253b9de-tls-certs\") pod \"kube-auth-proxy-58d577dc79-jtwnc\" (UID: \"0196e473-3f42-4506-bedd-da006253b9de\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:39.129156 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.129018    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0196e473-3f42-4506-bedd-da006253b9de-tmp\") pod \"kube-auth-proxy-58d577dc79-jtwnc\" (UID: \"0196e473-3f42-4506-bedd-da006253b9de\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:39.129156 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.129041    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bb4\" (UniqueName: \"kubernetes.io/projected/0196e473-3f42-4506-bedd-da006253b9de-kube-api-access-q5bb4\") pod \"kube-auth-proxy-58d577dc79-jtwnc\" (UID: \"0196e473-3f42-4506-bedd-da006253b9de\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:39.131169 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.131139    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0196e473-3f42-4506-bedd-da006253b9de-tmp\") pod \"kube-auth-proxy-58d577dc79-jtwnc\" (UID: \"0196e473-3f42-4506-bedd-da006253b9de\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:39.131395 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.131375    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0196e473-3f42-4506-bedd-da006253b9de-tls-certs\") pod \"kube-auth-proxy-58d577dc79-jtwnc\" (UID: \"0196e473-3f42-4506-bedd-da006253b9de\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:39.184242 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.184192    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bb4\" (UniqueName: \"kubernetes.io/projected/0196e473-3f42-4506-bedd-da006253b9de-kube-api-access-q5bb4\") pod \"kube-auth-proxy-58d577dc79-jtwnc\" (UID: \"0196e473-3f42-4506-bedd-da006253b9de\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:39.262379 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.262290    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"
Oct 30 18:41:39.627707 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:39.627624    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc"]
Oct 30 18:41:39.630025 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:41:39.630000    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0196e473_3f42_4506_bedd_da006253b9de.slice/crio-0bb7718ff86e16c7370735d0c69721daab37c5d970517c0a79f1469e825ea759 WatchSource:0}: Error finding container 0bb7718ff86e16c7370735d0c69721daab37c5d970517c0a79f1469e825ea759: Status 404 returned error can't find the container with id 0bb7718ff86e16c7370735d0c69721daab37c5d970517c0a79f1469e825ea759
Oct 30 18:41:40.308739 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:40.308698    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc" event={"ID":"0196e473-3f42-4506-bedd-da006253b9de","Type":"ContainerStarted","Data":"0bb7718ff86e16c7370735d0c69721daab37c5d970517c0a79f1469e825ea759"}
Oct 30 18:41:41.258166 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:41:41.258124    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:41:43.324552 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:43.324515    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc" event={"ID":"0196e473-3f42-4506-bedd-da006253b9de","Type":"ContainerStarted","Data":"83d29b9f1eb9397f0e6791530688472127a6ee5036ec0f24227b5ea4229c1ab1"}
Oct 30 18:41:43.433301 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:43.433244    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58d577dc79-jtwnc" podStartSLOduration=2.538858755 podStartE2EDuration="5.433231096s" totalImagesPullingTime="2.894372341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:43.432709949 +0000 UTC m=+298.543443233" watchObservedRunningTime="2026-10-30 18:41:43.433231096 +0000 UTC m=+298.543964378"
Oct 30 18:41:47.371238 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.371200    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"]
Oct 30 18:41:47.376072 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.376050    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.425755 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.425721    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 18:41:47.438139 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.438109    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-ftdxw\""
Oct 30 18:41:47.473770 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.473744    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"]
Oct 30 18:41:47.499371 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.499336    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.499487 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.499378    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.499487 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.499409    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.499599 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.499522    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.499655 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.499598    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76swf\" (UniqueName: \"kubernetes.io/projected/59262255-6bf4-49dc-86c2-db46d56acc3f-kube-api-access-76swf\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.499655 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.499636    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.499736 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.499670    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59262255-6bf4-49dc-86c2-db46d56acc3f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.499819 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.499801    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.499876 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.499838    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.600832 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.600798    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.600832 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.600834    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601058 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.600852    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601058 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.600882    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601058 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.600900    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76swf\" (UniqueName: \"kubernetes.io/projected/59262255-6bf4-49dc-86c2-db46d56acc3f-kube-api-access-76swf\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601058 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.600937    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601252 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.601097    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59262255-6bf4-49dc-86c2-db46d56acc3f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601252 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.601160    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601252 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.601200    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601397 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.601252    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601397 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.601285    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601397 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.601318    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601543 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.601472    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.601883 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.601863    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59262255-6bf4-49dc-86c2-db46d56acc3f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.603227 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.603208    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.603365 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.603344    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.656099 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.656072    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59262255-6bf4-49dc-86c2-db46d56acc3f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.656257 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.656126    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76swf\" (UniqueName: \"kubernetes.io/projected/59262255-6bf4-49dc-86c2-db46d56acc3f-kube-api-access-76swf\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj\" (UID: \"59262255-6bf4-49dc-86c2-db46d56acc3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:47.689030 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:47.688993    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:48.166527 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.166488    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"]
Oct 30 18:41:48.169916 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:41:48.169888    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59262255_6bf4_49dc_86c2_db46d56acc3f.slice/crio-3ce676bd726753d255b7cef3318d9da61cf2b43d81bd472e1529b09965bd9b81 WatchSource:0}: Error finding container 3ce676bd726753d255b7cef3318d9da61cf2b43d81bd472e1529b09965bd9b81: Status 404 returned error can't find the container with id 3ce676bd726753d255b7cef3318d9da61cf2b43d81bd472e1529b09965bd9b81
Oct 30 18:41:48.171884 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.171868    4391 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:41:48.346506 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.346469    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj" event={"ID":"59262255-6bf4-49dc-86c2-db46d56acc3f","Type":"ContainerStarted","Data":"3ce676bd726753d255b7cef3318d9da61cf2b43d81bd472e1529b09965bd9b81"}
Oct 30 18:41:48.375883 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.375856    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-r55t4"]
Oct 30 18:41:48.380611 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.380591    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:48.431760 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.431668    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 30 18:41:48.431760 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.431699    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-qg6fp\""
Oct 30 18:41:48.433758 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.433735    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-r55t4"]
Oct 30 18:41:48.510461 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.510427    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/34aaf976-fadc-4922-878c-626c5841487b-webhook-certs\") pod \"odh-observability-584f99c788-r55t4\" (UID: \"34aaf976-fadc-4922-878c-626c5841487b\") " pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:48.510615 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.510474    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdq95\" (UniqueName: \"kubernetes.io/projected/34aaf976-fadc-4922-878c-626c5841487b-kube-api-access-kdq95\") pod \"odh-observability-584f99c788-r55t4\" (UID: \"34aaf976-fadc-4922-878c-626c5841487b\") " pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:48.611480 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.611442    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/34aaf976-fadc-4922-878c-626c5841487b-webhook-certs\") pod \"odh-observability-584f99c788-r55t4\" (UID: \"34aaf976-fadc-4922-878c-626c5841487b\") " pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:48.611662 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.611500    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdq95\" (UniqueName: \"kubernetes.io/projected/34aaf976-fadc-4922-878c-626c5841487b-kube-api-access-kdq95\") pod \"odh-observability-584f99c788-r55t4\" (UID: \"34aaf976-fadc-4922-878c-626c5841487b\") " pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:48.611662 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:41:48.611635    4391 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 18:41:48.611767 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:41:48.611700    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34aaf976-fadc-4922-878c-626c5841487b-webhook-certs podName:34aaf976-fadc-4922-878c-626c5841487b nodeName:}" failed. No retries permitted until 2026-10-30 18:41:49.111684884 +0000 UTC m=+304.222418145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/34aaf976-fadc-4922-878c-626c5841487b-webhook-certs") pod "odh-observability-584f99c788-r55t4" (UID: "34aaf976-fadc-4922-878c-626c5841487b") : secret "odh-observability-webhook-cert" not found
Oct 30 18:41:48.668784 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:48.668755    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdq95\" (UniqueName: \"kubernetes.io/projected/34aaf976-fadc-4922-878c-626c5841487b-kube-api-access-kdq95\") pod \"odh-observability-584f99c788-r55t4\" (UID: \"34aaf976-fadc-4922-878c-626c5841487b\") " pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:49.115568 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:49.115535    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/34aaf976-fadc-4922-878c-626c5841487b-webhook-certs\") pod \"odh-observability-584f99c788-r55t4\" (UID: \"34aaf976-fadc-4922-878c-626c5841487b\") " pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:49.115777 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:41:49.115721    4391 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 30 18:41:49.115847 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:41:49.115801    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34aaf976-fadc-4922-878c-626c5841487b-webhook-certs podName:34aaf976-fadc-4922-878c-626c5841487b nodeName:}" failed. No retries permitted until 2026-10-30 18:41:50.115779881 +0000 UTC m=+305.226513159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/34aaf976-fadc-4922-878c-626c5841487b-webhook-certs") pod "odh-observability-584f99c788-r55t4" (UID: "34aaf976-fadc-4922-878c-626c5841487b") : secret "odh-observability-webhook-cert" not found
Oct 30 18:41:50.124525 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:50.124485    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/34aaf976-fadc-4922-878c-626c5841487b-webhook-certs\") pod \"odh-observability-584f99c788-r55t4\" (UID: \"34aaf976-fadc-4922-878c-626c5841487b\") " pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:50.127236 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:50.127209    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/34aaf976-fadc-4922-878c-626c5841487b-webhook-certs\") pod \"odh-observability-584f99c788-r55t4\" (UID: \"34aaf976-fadc-4922-878c-626c5841487b\") " pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:50.193344 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:50.193309    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:50.581421 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:50.581377    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-r55t4"]
Oct 30 18:41:50.583927 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:41:50.583896    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34aaf976_fadc_4922_878c_626c5841487b.slice/crio-9980d5257e111094f3f2c8509eb8595c0b782f375132b00cd22a67924bce6961 WatchSource:0}: Error finding container 9980d5257e111094f3f2c8509eb8595c0b782f375132b00cd22a67924bce6961: Status 404 returned error can't find the container with id 9980d5257e111094f3f2c8509eb8595c0b782f375132b00cd22a67924bce6961
Oct 30 18:41:51.366600 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:51.366538    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-r55t4" event={"ID":"34aaf976-fadc-4922-878c-626c5841487b","Type":"ContainerStarted","Data":"9980d5257e111094f3f2c8509eb8595c0b782f375132b00cd22a67924bce6961"}
Oct 30 18:41:52.136753 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:52.136724    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:41:52.136855 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:52.136807    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:41:52.136855 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:52.136834    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:41:52.375278 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:52.374778    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj" event={"ID":"59262255-6bf4-49dc-86c2-db46d56acc3f","Type":"ContainerStarted","Data":"b558c0f028d837957a3ea20d51dca4814756e2a165d092219c867c5154f8b495"}
Oct 30 18:41:52.534996 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:52.534938    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj" podStartSLOduration=1.5710823600000001 podStartE2EDuration="5.53492247s" totalImagesPullingTime="3.96384011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:52.534300005 +0000 UTC m=+307.645033288" watchObservedRunningTime="2026-10-30 18:41:52.53492247 +0000 UTC m=+307.645655755"
Oct 30 18:41:52.689708 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:52.689671    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:53.694526 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:53.694492    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:54.385665 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:54.385633    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:54.386801 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:54.386781    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj"
Oct 30 18:41:55.391458 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:55.391417    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-r55t4" event={"ID":"34aaf976-fadc-4922-878c-626c5841487b","Type":"ContainerStarted","Data":"97a2cb589911953662afbab01388774fbfc6d2f9c9a9b438f61ecfba8f57e3a1"}
Oct 30 18:41:55.391965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:55.391504    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:41:55.500878 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:41:55.500821    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-r55t4" podStartSLOduration=3.578763329 podStartE2EDuration="7.50080588s" totalImagesPullingTime="3.922042551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:55.5002683 +0000 UTC m=+310.611001584" watchObservedRunningTime="2026-10-30 18:41:55.50080588 +0000 UTC m=+310.611539163"
Oct 30 18:42:06.099527 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.099426    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"]
Oct 30 18:42:06.103294 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.103273    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.165253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.165220    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6wmp\" (UniqueName: \"kubernetes.io/projected/9bbf302c-76be-47e8-84fd-1cc70e3f99df-kube-api-access-d6wmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.165406 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.165260    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bbf302c-76be-47e8-84fd-1cc70e3f99df-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.165406 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.165365    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9bbf302c-76be-47e8-84fd-1cc70e3f99df-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.165406 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.165388    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bbf302c-76be-47e8-84fd-1cc70e3f99df-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.200048 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.200012    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-8wwqb\""
Oct 30 18:42:06.266015 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.265983    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bbf302c-76be-47e8-84fd-1cc70e3f99df-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.266191 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.266072    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9bbf302c-76be-47e8-84fd-1cc70e3f99df-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.266191 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.266102    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bbf302c-76be-47e8-84fd-1cc70e3f99df-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.266191 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.266164    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6wmp\" (UniqueName: \"kubernetes.io/projected/9bbf302c-76be-47e8-84fd-1cc70e3f99df-kube-api-access-d6wmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.266447 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.266423    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bbf302c-76be-47e8-84fd-1cc70e3f99df-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.266547 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.266529    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9bbf302c-76be-47e8-84fd-1cc70e3f99df-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.294203 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.294173    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 18:42:06.295871 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.295853    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"]
Oct 30 18:42:06.298311 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.298290    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9bbf302c-76be-47e8-84fd-1cc70e3f99df-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.320716 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.320685    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6wmp\" (UniqueName: \"kubernetes.io/projected/9bbf302c-76be-47e8-84fd-1cc70e3f99df-kube-api-access-d6wmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-cjtr8\" (UID: \"9bbf302c-76be-47e8-84fd-1cc70e3f99df\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.398415 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.398337    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-r55t4"
Oct 30 18:42:06.415971 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.415949    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:06.831687 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:06.831575    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"]
Oct 30 18:42:06.834282 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:42:06.834248    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbf302c_76be_47e8_84fd_1cc70e3f99df.slice/crio-afd13a761ae5b6880337d56e3291567c6dfa027c18283ea0d73791119ae3ee9f WatchSource:0}: Error finding container afd13a761ae5b6880337d56e3291567c6dfa027c18283ea0d73791119ae3ee9f: Status 404 returned error can't find the container with id afd13a761ae5b6880337d56e3291567c6dfa027c18283ea0d73791119ae3ee9f
Oct 30 18:42:07.444590 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:07.444539    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8" event={"ID":"9bbf302c-76be-47e8-84fd-1cc70e3f99df","Type":"ContainerStarted","Data":"afd13a761ae5b6880337d56e3291567c6dfa027c18283ea0d73791119ae3ee9f"}
Oct 30 18:42:10.460472 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:10.460437    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8" event={"ID":"9bbf302c-76be-47e8-84fd-1cc70e3f99df","Type":"ContainerStarted","Data":"143d58d2d44fa34047fa7c08a224dd80dcdbc4a5e05331c5ab493dbfd90cc1df"}
Oct 30 18:42:10.460884 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:10.460554    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:10.618464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:10.618410    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8" podStartSLOduration=1.859948521 podStartE2EDuration="4.618394064s" totalImagesPullingTime="2.758445543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:10.617353462 +0000 UTC m=+325.728086745" watchObservedRunningTime="2026-10-30 18:42:10.618394064 +0000 UTC m=+325.729127347"
Oct 30 18:42:11.252333 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:42:11.252296    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:42:31.465416 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:31.465385    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-cjtr8"
Oct 30 18:42:41.252185 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:42:41.252142    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:42:57.521403 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.521359    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"]
Oct 30 18:42:57.533275 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.533246    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.584674 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.584645    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 18:42:57.584966 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.584947    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 18:42:57.585064 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.585040    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 18:42:57.630260 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.630234    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ntm7n\""
Oct 30 18:42:57.632100 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.632080    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"]
Oct 30 18:42:57.710835 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.710799    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.710835 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.710834    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfph8\" (UniqueName: \"kubernetes.io/projected/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-kube-api-access-gfph8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.711076 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.710860    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.711076 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.710942    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.711076 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.711016    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.711076 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.711067    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.711255 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.711113    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.812492 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.812411    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.812492 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.812448    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfph8\" (UniqueName: \"kubernetes.io/projected/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-kube-api-access-gfph8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.812492 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.812482    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.812796 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.812504    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.812796 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.812537    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.812796 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.812575    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.812796 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.812640    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.813433 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.813407    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.815018 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.814992    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.815106 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.814997    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.815268 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.815252    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.815304 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.815287    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.868823 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.868798    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:57.869126 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:57.869102    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfph8\" (UniqueName: \"kubernetes.io/projected/a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25-kube-api-access-gfph8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27pn2\" (UID: \"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:58.145131 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:58.145052    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:42:58.609419 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:58.609395    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"]
Oct 30 18:42:58.614326 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:42:58.612761    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a814f8_0aeb_4ba5_895e_8fbf7ed57d25.slice/crio-c1b3a6bd018bd7bf91d6a630d6e10becbefed2fc0124b39d7f35eb55821198a6 WatchSource:0}: Error finding container c1b3a6bd018bd7bf91d6a630d6e10becbefed2fc0124b39d7f35eb55821198a6: Status 404 returned error can't find the container with id c1b3a6bd018bd7bf91d6a630d6e10becbefed2fc0124b39d7f35eb55821198a6
Oct 30 18:42:58.670801 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:42:58.670770    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2" event={"ID":"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25","Type":"ContainerStarted","Data":"c1b3a6bd018bd7bf91d6a630d6e10becbefed2fc0124b39d7f35eb55821198a6"}
Oct 30 18:43:02.937606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:02.937542    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-gmjrz"]
Oct 30 18:43:02.945999 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:02.945968    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz"
Oct 30 18:43:02.952882 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:02.952846    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ad60ab5-d7d1-4857-9df0-645b4bdc607d-cert\") pod \"kserve-controller-manager-776cc66d47-gmjrz\" (UID: \"8ad60ab5-d7d1-4857-9df0-645b4bdc607d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz"
Oct 30 18:43:02.953033 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:02.952889    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b9ml\" (UniqueName: \"kubernetes.io/projected/8ad60ab5-d7d1-4857-9df0-645b4bdc607d-kube-api-access-6b9ml\") pod \"kserve-controller-manager-776cc66d47-gmjrz\" (UID: \"8ad60ab5-d7d1-4857-9df0-645b4bdc607d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz"
Oct 30 18:43:02.994576 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:02.994536    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-gmjrz"]
Oct 30 18:43:03.038754 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.038717    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gncqx\""
Oct 30 18:43:03.042357 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.042335    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 18:43:03.044323 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.044302    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"]
Oct 30 18:43:03.047966 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.047948    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:43:03.053487 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.053462    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ad60ab5-d7d1-4857-9df0-645b4bdc607d-cert\") pod \"kserve-controller-manager-776cc66d47-gmjrz\" (UID: \"8ad60ab5-d7d1-4857-9df0-645b4bdc607d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz"
Oct 30 18:43:03.053487 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.053499    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b9ml\" (UniqueName: \"kubernetes.io/projected/8ad60ab5-d7d1-4857-9df0-645b4bdc607d-kube-api-access-6b9ml\") pod \"kserve-controller-manager-776cc66d47-gmjrz\" (UID: \"8ad60ab5-d7d1-4857-9df0-645b4bdc607d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz"
Oct 30 18:43:03.053740 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.053559    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thbdj\" (UniqueName: \"kubernetes.io/projected/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-kube-api-access-thbdj\") pod \"llmisvc-controller-manager-869d5dcc5d-ssfzq\" (UID: \"fecaeae5-cc1d-40ad-a6dc-923863e99ed5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:43:03.053740 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.053615    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ssfzq\" (UID: \"fecaeae5-cc1d-40ad-a6dc-923863e99ed5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:43:03.057686 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.057662    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ad60ab5-d7d1-4857-9df0-645b4bdc607d-cert\") pod \"kserve-controller-manager-776cc66d47-gmjrz\" (UID: \"8ad60ab5-d7d1-4857-9df0-645b4bdc607d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz"
Oct 30 18:43:03.092610 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.092558    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"]
Oct 30 18:43:03.098219 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.098192    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 18:43:03.098219 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.098210    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2gkgb\""
Oct 30 18:43:03.109706 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.109649    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b9ml\" (UniqueName: \"kubernetes.io/projected/8ad60ab5-d7d1-4857-9df0-645b4bdc607d-kube-api-access-6b9ml\") pod \"kserve-controller-manager-776cc66d47-gmjrz\" (UID: \"8ad60ab5-d7d1-4857-9df0-645b4bdc607d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz"
Oct 30 18:43:03.145249 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.145212    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jtmwb"]
Oct 30 18:43:03.149022 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.148995    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb"
Oct 30 18:43:03.155083 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.155053    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnxw\" (UniqueName: \"kubernetes.io/projected/8a6fb2a0-b06b-4a31-b663-2e7ec491a21a-kube-api-access-mvnxw\") pod \"model-serving-api-75bd89fdfc-jtmwb\" (UID: \"8a6fb2a0-b06b-4a31-b663-2e7ec491a21a\") " pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb"
Oct 30 18:43:03.155217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.155107    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thbdj\" (UniqueName: \"kubernetes.io/projected/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-kube-api-access-thbdj\") pod \"llmisvc-controller-manager-869d5dcc5d-ssfzq\" (UID: \"fecaeae5-cc1d-40ad-a6dc-923863e99ed5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:43:03.155217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.155200    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ssfzq\" (UID: \"fecaeae5-cc1d-40ad-a6dc-923863e99ed5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:43:03.155350 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.155326    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6fb2a0-b06b-4a31-b663-2e7ec491a21a-tls-certs\") pod \"model-serving-api-75bd89fdfc-jtmwb\" (UID: \"8a6fb2a0-b06b-4a31-b663-2e7ec491a21a\") " pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb"
Oct 30 18:43:03.158152 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.158125    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ssfzq\" (UID: \"fecaeae5-cc1d-40ad-a6dc-923863e99ed5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:43:03.199560 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.199481    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-btvfk\""
Oct 30 18:43:03.199749 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.199661    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 18:43:03.205605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.205552    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jtmwb"]
Oct 30 18:43:03.210102 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.210050    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thbdj\" (UniqueName: \"kubernetes.io/projected/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-kube-api-access-thbdj\") pod \"llmisvc-controller-manager-869d5dcc5d-ssfzq\" (UID: \"fecaeae5-cc1d-40ad-a6dc-923863e99ed5\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:43:03.251994 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.251961    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-4dltq"]
Oct 30 18:43:03.256235 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.256210    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6fb2a0-b06b-4a31-b663-2e7ec491a21a-tls-certs\") pod \"model-serving-api-75bd89fdfc-jtmwb\" (UID: \"8a6fb2a0-b06b-4a31-b663-2e7ec491a21a\") " pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb"
Oct 30 18:43:03.256454 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.256284    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnxw\" (UniqueName: \"kubernetes.io/projected/8a6fb2a0-b06b-4a31-b663-2e7ec491a21a-kube-api-access-mvnxw\") pod \"model-serving-api-75bd89fdfc-jtmwb\" (UID: \"8a6fb2a0-b06b-4a31-b663-2e7ec491a21a\") " pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb"
Oct 30 18:43:03.258726 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.258688    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.259088 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.259052    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6fb2a0-b06b-4a31-b663-2e7ec491a21a-tls-certs\") pod \"model-serving-api-75bd89fdfc-jtmwb\" (UID: \"8a6fb2a0-b06b-4a31-b663-2e7ec491a21a\") " pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb"
Oct 30 18:43:03.263429 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.263406    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz"
Oct 30 18:43:03.308795 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.308617    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 18:43:03.308795 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.308711    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gmmqs\""
Oct 30 18:43:03.308795 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.308761    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 18:43:03.311348 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.311310    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-4dltq"]
Oct 30 18:43:03.314290 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.314265    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnxw\" (UniqueName: \"kubernetes.io/projected/8a6fb2a0-b06b-4a31-b663-2e7ec491a21a-kube-api-access-mvnxw\") pod \"model-serving-api-75bd89fdfc-jtmwb\" (UID: \"8a6fb2a0-b06b-4a31-b663-2e7ec491a21a\") " pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb"
Oct 30 18:43:03.357614 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.357545    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd502785-059e-4fc7-acf0-af91400f840e-metrics-certs\") pod \"odh-model-controller-f87fcf56c-4dltq\" (UID: \"cd502785-059e-4fc7-acf0-af91400f840e\") " pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.357793 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.357693    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd502785-059e-4fc7-acf0-af91400f840e-cert\") pod \"odh-model-controller-f87fcf56c-4dltq\" (UID: \"cd502785-059e-4fc7-acf0-af91400f840e\") " pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.357793 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.357770    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj5dk\" (UniqueName: \"kubernetes.io/projected/cd502785-059e-4fc7-acf0-af91400f840e-kube-api-access-fj5dk\") pod \"odh-model-controller-f87fcf56c-4dltq\" (UID: \"cd502785-059e-4fc7-acf0-af91400f840e\") " pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.371673 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.371515    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:43:03.459414 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.459335    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd502785-059e-4fc7-acf0-af91400f840e-cert\") pod \"odh-model-controller-f87fcf56c-4dltq\" (UID: \"cd502785-059e-4fc7-acf0-af91400f840e\") " pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.459558 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.459482    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj5dk\" (UniqueName: \"kubernetes.io/projected/cd502785-059e-4fc7-acf0-af91400f840e-kube-api-access-fj5dk\") pod \"odh-model-controller-f87fcf56c-4dltq\" (UID: \"cd502785-059e-4fc7-acf0-af91400f840e\") " pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.459558 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.459526    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd502785-059e-4fc7-acf0-af91400f840e-metrics-certs\") pod \"odh-model-controller-f87fcf56c-4dltq\" (UID: \"cd502785-059e-4fc7-acf0-af91400f840e\") " pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.461921 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.461892    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd502785-059e-4fc7-acf0-af91400f840e-cert\") pod \"odh-model-controller-f87fcf56c-4dltq\" (UID: \"cd502785-059e-4fc7-acf0-af91400f840e\") " pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.462033 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.461898    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd502785-059e-4fc7-acf0-af91400f840e-metrics-certs\") pod \"odh-model-controller-f87fcf56c-4dltq\" (UID: \"cd502785-059e-4fc7-acf0-af91400f840e\") " pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.473721 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.473696    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb"
Oct 30 18:43:03.488421 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.488393    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:43:03.488498 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.488452    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:43:03.514938 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.514908    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj5dk\" (UniqueName: \"kubernetes.io/projected/cd502785-059e-4fc7-acf0-af91400f840e-kube-api-access-fj5dk\") pod \"odh-model-controller-f87fcf56c-4dltq\" (UID: \"cd502785-059e-4fc7-acf0-af91400f840e\") " pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.584257 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.584223    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:03.641800 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.641764    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-gmjrz"]
Oct 30 18:43:03.697997 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.697946    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2" event={"ID":"a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25","Type":"ContainerStarted","Data":"4ace8264f5da77cf7c1c3555e9b4e089de54ec4a9cb9b7998ef418b4e7560b5a"}
Oct 30 18:43:03.698188 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.698079    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:43:03.699169 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.699143    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz" event={"ID":"8ad60ab5-d7d1-4857-9df0-645b4bdc607d","Type":"ContainerStarted","Data":"ef8f74c580c1a93ad71aeb2f868d7d13da1d445457455fe37f6d5a9256cd41ed"}
Oct 30 18:43:03.749516 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.749475    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"]
Oct 30 18:43:03.752391 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:43:03.752356    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfecaeae5_cc1d_40ad_a6dc_923863e99ed5.slice/crio-1984a2dc03e4b1d45c7f54d0030daadfc337f49ef565c5f2ebcc0009bc309e96 WatchSource:0}: Error finding container 1984a2dc03e4b1d45c7f54d0030daadfc337f49ef565c5f2ebcc0009bc309e96: Status 404 returned error can't find the container with id 1984a2dc03e4b1d45c7f54d0030daadfc337f49ef565c5f2ebcc0009bc309e96
Oct 30 18:43:03.828602 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.828539    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2" podStartSLOduration=1.957560568 podStartE2EDuration="6.828524091s" totalImagesPullingTime="4.870963523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:03.827449475 +0000 UTC m=+378.938182758" watchObservedRunningTime="2026-10-30 18:43:03.828524091 +0000 UTC m=+378.939257373"
Oct 30 18:43:03.849051 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.849020    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jtmwb"]
Oct 30 18:43:03.852034 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:43:03.851997    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6fb2a0_b06b_4a31_b663_2e7ec491a21a.slice/crio-6ea5260d6fd9455b059e7f8899c0bcebadc703c9ea4d8c279bd50ee8ef2cac9d WatchSource:0}: Error finding container 6ea5260d6fd9455b059e7f8899c0bcebadc703c9ea4d8c279bd50ee8ef2cac9d: Status 404 returned error can't find the container with id 6ea5260d6fd9455b059e7f8899c0bcebadc703c9ea4d8c279bd50ee8ef2cac9d
Oct 30 18:43:03.953018 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:03.952861    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-4dltq"]
Oct 30 18:43:03.955126 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:43:03.955093    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd502785_059e_4fc7_acf0_af91400f840e.slice/crio-c623e3f34029b74dbf5494b6a62390c7fc29ae91a8569c7cbdd58d1c4f28cdca WatchSource:0}: Error finding container c623e3f34029b74dbf5494b6a62390c7fc29ae91a8569c7cbdd58d1c4f28cdca: Status 404 returned error can't find the container with id c623e3f34029b74dbf5494b6a62390c7fc29ae91a8569c7cbdd58d1c4f28cdca
Oct 30 18:43:04.722067 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:04.721957    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-4dltq" event={"ID":"cd502785-059e-4fc7-acf0-af91400f840e","Type":"ContainerStarted","Data":"c623e3f34029b74dbf5494b6a62390c7fc29ae91a8569c7cbdd58d1c4f28cdca"}
Oct 30 18:43:04.723927 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:04.723891    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq" event={"ID":"fecaeae5-cc1d-40ad-a6dc-923863e99ed5","Type":"ContainerStarted","Data":"1984a2dc03e4b1d45c7f54d0030daadfc337f49ef565c5f2ebcc0009bc309e96"}
Oct 30 18:43:04.726292 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:04.726198    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb" event={"ID":"8a6fb2a0-b06b-4a31-b663-2e7ec491a21a","Type":"ContainerStarted","Data":"6ea5260d6fd9455b059e7f8899c0bcebadc703c9ea4d8c279bd50ee8ef2cac9d"}
Oct 30 18:43:04.729090 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:04.729060    4391 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-27pn2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:43:04.729213 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:04.729122    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2" podUID="a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:43:07.727441 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:07.727416    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27pn2"
Oct 30 18:43:07.748644 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:07.748610    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb" event={"ID":"8a6fb2a0-b06b-4a31-b663-2e7ec491a21a","Type":"ContainerStarted","Data":"27495512d7eb3245ca4d2aaf683d4b144a61167be59dd0dd00b1dc99eb4f944a"}
Oct 30 18:43:07.748783 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:07.748761    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb"
Oct 30 18:43:07.750543 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:07.750519    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz" event={"ID":"8ad60ab5-d7d1-4857-9df0-645b4bdc607d","Type":"ContainerStarted","Data":"bf0b819fdb65e4465780fda82fbf91e16ff29eb197be08e19e5bace44aae7498"}
Oct 30 18:43:07.750967 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:07.750949    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz"
Oct 30 18:43:07.973441 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:07.973307    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz" podStartSLOduration=2.503412945 podStartE2EDuration="5.973292405s" totalImagesPullingTime="3.46987946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:07.970304945 +0000 UTC m=+383.081038231" watchObservedRunningTime="2026-10-30 18:43:07.973292405 +0000 UTC m=+383.084025687"
Oct 30 18:43:08.078753 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:08.078690    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb" podStartSLOduration=2.141975123 podStartE2EDuration="5.078670301s" totalImagesPullingTime="2.936695178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:08.078010691 +0000 UTC m=+383.188743972" watchObservedRunningTime="2026-10-30 18:43:08.078670301 +0000 UTC m=+383.189403587"
Oct 30 18:43:08.764902 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:08.764864    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-4dltq" event={"ID":"cd502785-059e-4fc7-acf0-af91400f840e","Type":"ContainerStarted","Data":"e9480ec1120f981e6ae489481566f1664f38ea8e0908ebf11325e59c96f9f1df"}
Oct 30 18:43:08.765372 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:08.764951    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:08.766400 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:08.766366    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq" event={"ID":"fecaeae5-cc1d-40ad-a6dc-923863e99ed5","Type":"ContainerStarted","Data":"4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d"}
Oct 30 18:43:08.766635 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:08.766612    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:43:08.874397 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:08.874344    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-4dltq" podStartSLOduration=2.128427396 podStartE2EDuration="5.87433203s" totalImagesPullingTime="3.745904634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:08.872698734 +0000 UTC m=+383.983432017" watchObservedRunningTime="2026-10-30 18:43:08.87433203 +0000 UTC m=+383.985065312"
Oct 30 18:43:08.982537 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:08.982486    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq" podStartSLOduration=2.432767392 podStartE2EDuration="6.982471193s" totalImagesPullingTime="4.549703801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:08.981741281 +0000 UTC m=+384.092474563" watchObservedRunningTime="2026-10-30 18:43:08.982471193 +0000 UTC m=+384.093204476"
Oct 30 18:43:11.253140 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:43:11.253101    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:43:18.770418 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:18.770385    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-jtmwb"
Oct 30 18:43:19.773707 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:19.773680    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-4dltq"
Oct 30 18:43:39.774112 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:39.774082    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:43:39.777002 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:43:39.776985    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-gmjrz"
Oct 30 18:43:41.252462 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:43:41.252421    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:44:11.252708 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:44:11.252673    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:44:13.300030 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.299997    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"]
Oct 30 18:44:13.303254 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.303230    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.402120 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.402087    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:44:13.402336 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.402169    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-xqzfz\""
Oct 30 18:44:13.402447 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.402366    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:44:13.404258 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.404233    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"]
Oct 30 18:44:13.405954 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.405923    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"]
Oct 30 18:44:13.409018 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.408998    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.415546 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.415519    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.415790 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.415662    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8c4\" (UniqueName: \"kubernetes.io/projected/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-kube-api-access-6x8c4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.415859 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.415787    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.459282 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.459249    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-kktsk\""
Oct 30 18:44:13.461091 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.461070    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"]
Oct 30 18:44:13.500412 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.499974    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"]
Oct 30 18:44:13.506230 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.506206    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.516272 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.516249    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.516414 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.516397    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfz8b\" (UniqueName: \"kubernetes.io/projected/b0701727-b74d-4156-a436-bad574f82272-kube-api-access-qfz8b\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.516494 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.516440    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.516494 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.516476    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.516613 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.516593    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.516665 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.516614    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8c4\" (UniqueName: \"kubernetes.io/projected/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-kube-api-access-6x8c4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.516665 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.516657    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.516821 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.516795    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.556144 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.556075    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"]
Oct 30 18:44:13.559259 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.559229    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-fszrw\""
Oct 30 18:44:13.570929 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.570901    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8c4\" (UniqueName: \"kubernetes.io/projected/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-kube-api-access-6x8c4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.600048 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.600022    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"]
Oct 30 18:44:13.603062 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.603044    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.617272 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.617250    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfz8b\" (UniqueName: \"kubernetes.io/projected/b0701727-b74d-4156-a436-bad574f82272-kube-api-access-qfz8b\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.617357 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.617294    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.617357 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.617318    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ztd4\" (UniqueName: \"kubernetes.io/projected/5408995b-40ff-4f01-acc2-e8d8b690fc4a-kube-api-access-4ztd4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.617455 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.617359    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.617455 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.617385    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.617556 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.617518    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.617556 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.617525    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:13.617789 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.617689    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.617789 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.617755    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.658147 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.658109    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"]
Oct 30 18:44:13.660128 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.660106    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-jmmrc\""
Oct 30 18:44:13.680346 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.680310    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfz8b\" (UniqueName: \"kubernetes.io/projected/b0701727-b74d-4156-a436-bad574f82272-kube-api-access-qfz8b\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.718940 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.718910    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkh2g\" (UniqueName: \"kubernetes.io/projected/12ccb2f4-40ab-491a-b839-99b7c921442c-kube-api-access-xkh2g\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.719093 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.718965    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.719093 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.719052    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.719224 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.719112    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.719224 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.719164    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.719224 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.719208    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztd4\" (UniqueName: \"kubernetes.io/projected/5408995b-40ff-4f01-acc2-e8d8b690fc4a-kube-api-access-4ztd4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.719487 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.719466    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.719553 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.719496    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.723772 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.723748    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:13.778313 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.778281    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztd4\" (UniqueName: \"kubernetes.io/projected/5408995b-40ff-4f01-acc2-e8d8b690fc4a-kube-api-access-4ztd4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.820453 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.820387    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:13.820695 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.820670    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.820758 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.820716    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.820810 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.820797    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkh2g\" (UniqueName: \"kubernetes.io/projected/12ccb2f4-40ab-491a-b839-99b7c921442c-kube-api-access-xkh2g\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.821037 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.821018    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.821075 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.821049    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.878726 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.878696    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkh2g\" (UniqueName: \"kubernetes.io/projected/12ccb2f4-40ab-491a-b839-99b7c921442c-kube-api-access-xkh2g\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.917519 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.917488    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:13.992564 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:13.992533    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"]
Oct 30 18:44:13.993194 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:44:13.993142    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba1ad2a_abbd_4976_ad83_4f76b7bc10c7.slice/crio-73c42e70fcf11b88d02621a2af03f4393ae3c04852e66d80349f7b54673598ea WatchSource:0}: Error finding container 73c42e70fcf11b88d02621a2af03f4393ae3c04852e66d80349f7b54673598ea: Status 404 returned error can't find the container with id 73c42e70fcf11b88d02621a2af03f4393ae3c04852e66d80349f7b54673598ea
Oct 30 18:44:14.078684 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:14.078638    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk" event={"ID":"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7","Type":"ContainerStarted","Data":"c29469d041c8ed34c4da07ffd278c60dc0a24a33b9b469ce4bbaa415bfabb0ef"}
Oct 30 18:44:14.078800 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:14.078712    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk" event={"ID":"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7","Type":"ContainerStarted","Data":"73c42e70fcf11b88d02621a2af03f4393ae3c04852e66d80349f7b54673598ea"}
Oct 30 18:44:14.090248 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:14.090227    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"]
Oct 30 18:44:14.145345 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:44:14.145312    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0701727_b74d_4156_a436_bad574f82272.slice/crio-b2cabf6f29616969ba0f30aa7aea36ffdc98935d726eda8c4db011dbef01b900 WatchSource:0}: Error finding container b2cabf6f29616969ba0f30aa7aea36ffdc98935d726eda8c4db011dbef01b900: Status 404 returned error can't find the container with id b2cabf6f29616969ba0f30aa7aea36ffdc98935d726eda8c4db011dbef01b900
Oct 30 18:44:14.183520 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:14.183497    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"]
Oct 30 18:44:14.186111 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:44:14.186087    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5408995b_40ff_4f01_acc2_e8d8b690fc4a.slice/crio-076c534d88a045c00dfff2877df14067424ba0e180f88f8070f795cd569b3302 WatchSource:0}: Error finding container 076c534d88a045c00dfff2877df14067424ba0e180f88f8070f795cd569b3302: Status 404 returned error can't find the container with id 076c534d88a045c00dfff2877df14067424ba0e180f88f8070f795cd569b3302
Oct 30 18:44:14.284115 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:14.284088    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"]
Oct 30 18:44:14.286371 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:44:14.286330    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ccb2f4_40ab_491a_b839_99b7c921442c.slice/crio-f6bd0148dc0bb3ef0cb73ed4aeb4c04ef9ad900415335fd5d7cf46614d7139d0 WatchSource:0}: Error finding container f6bd0148dc0bb3ef0cb73ed4aeb4c04ef9ad900415335fd5d7cf46614d7139d0: Status 404 returned error can't find the container with id f6bd0148dc0bb3ef0cb73ed4aeb4c04ef9ad900415335fd5d7cf46614d7139d0
Oct 30 18:44:15.085009 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.084966    4391 generic.go:356] "Generic (PLEG): container finished" podID="5408995b-40ff-4f01-acc2-e8d8b690fc4a" containerID="ff54be318f5b11a941b75a6411d4552d747c57de39588646aa5f9375122f8cae" exitCode=0
Oct 30 18:44:15.085469 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.085031    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr" event={"ID":"5408995b-40ff-4f01-acc2-e8d8b690fc4a","Type":"ContainerDied","Data":"ff54be318f5b11a941b75a6411d4552d747c57de39588646aa5f9375122f8cae"}
Oct 30 18:44:15.085469 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.085078    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr" event={"ID":"5408995b-40ff-4f01-acc2-e8d8b690fc4a","Type":"ContainerStarted","Data":"076c534d88a045c00dfff2877df14067424ba0e180f88f8070f795cd569b3302"}
Oct 30 18:44:15.086522 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.086498    4391 generic.go:356] "Generic (PLEG): container finished" podID="b0701727-b74d-4156-a436-bad574f82272" containerID="f23e59d0792d394330a5e31010c20949dd84b87a111b7fa45fad9ec2eb53b5a9" exitCode=0
Oct 30 18:44:15.086632 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.086603    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr" event={"ID":"b0701727-b74d-4156-a436-bad574f82272","Type":"ContainerDied","Data":"f23e59d0792d394330a5e31010c20949dd84b87a111b7fa45fad9ec2eb53b5a9"}
Oct 30 18:44:15.086706 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.086648    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr" event={"ID":"b0701727-b74d-4156-a436-bad574f82272","Type":"ContainerStarted","Data":"b2cabf6f29616969ba0f30aa7aea36ffdc98935d726eda8c4db011dbef01b900"}
Oct 30 18:44:15.088416 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.088308    4391 generic.go:356] "Generic (PLEG): container finished" podID="12ccb2f4-40ab-491a-b839-99b7c921442c" containerID="4297fa2694ea20cc88b67cac7aa6061149e5de4c4a0c1afe53d90449f8650785" exitCode=0
Oct 30 18:44:15.088416 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.088384    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9" event={"ID":"12ccb2f4-40ab-491a-b839-99b7c921442c","Type":"ContainerDied","Data":"4297fa2694ea20cc88b67cac7aa6061149e5de4c4a0c1afe53d90449f8650785"}
Oct 30 18:44:15.088529 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.088425    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9" event={"ID":"12ccb2f4-40ab-491a-b839-99b7c921442c","Type":"ContainerStarted","Data":"f6bd0148dc0bb3ef0cb73ed4aeb4c04ef9ad900415335fd5d7cf46614d7139d0"}
Oct 30 18:44:15.090166 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.090094    4391 generic.go:356] "Generic (PLEG): container finished" podID="4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" containerID="c29469d041c8ed34c4da07ffd278c60dc0a24a33b9b469ce4bbaa415bfabb0ef" exitCode=0
Oct 30 18:44:15.090166 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:15.090149    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk" event={"ID":"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7","Type":"ContainerDied","Data":"c29469d041c8ed34c4da07ffd278c60dc0a24a33b9b469ce4bbaa415bfabb0ef"}
Oct 30 18:44:17.102219 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:17.102180    4391 generic.go:356] "Generic (PLEG): container finished" podID="b0701727-b74d-4156-a436-bad574f82272" containerID="5c62e13b891f8fa36f0dc6d32d4b39f47a2e67799b62ddd1cbb01fe64745dd84" exitCode=0
Oct 30 18:44:17.102634 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:17.102258    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr" event={"ID":"b0701727-b74d-4156-a436-bad574f82272","Type":"ContainerDied","Data":"5c62e13b891f8fa36f0dc6d32d4b39f47a2e67799b62ddd1cbb01fe64745dd84"}
Oct 30 18:44:17.107877 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:17.107824    4391 generic.go:356] "Generic (PLEG): container finished" podID="12ccb2f4-40ab-491a-b839-99b7c921442c" containerID="f08c88b4947470c65c378a17fda3aa05d1eb3785a0fa63f12740f4b83d04a53e" exitCode=0
Oct 30 18:44:17.107931 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:17.107862    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9" event={"ID":"12ccb2f4-40ab-491a-b839-99b7c921442c","Type":"ContainerDied","Data":"f08c88b4947470c65c378a17fda3aa05d1eb3785a0fa63f12740f4b83d04a53e"}
Oct 30 18:44:17.109998 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:17.109978    4391 generic.go:356] "Generic (PLEG): container finished" podID="4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" containerID="ec71294cef13901ec1a599c1828934250e27437a9eb7c031f393862c19435972" exitCode=0
Oct 30 18:44:17.110089 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:17.110017    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk" event={"ID":"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7","Type":"ContainerDied","Data":"ec71294cef13901ec1a599c1828934250e27437a9eb7c031f393862c19435972"}
Oct 30 18:44:17.111750 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:17.111731    4391 generic.go:356] "Generic (PLEG): container finished" podID="5408995b-40ff-4f01-acc2-e8d8b690fc4a" containerID="cbb4516bbf02fedf6ac9d3f1821e3e135646086ead4c5e044f758d6039a6b379" exitCode=0
Oct 30 18:44:17.111852 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:17.111761    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr" event={"ID":"5408995b-40ff-4f01-acc2-e8d8b690fc4a","Type":"ContainerDied","Data":"cbb4516bbf02fedf6ac9d3f1821e3e135646086ead4c5e044f758d6039a6b379"}
Oct 30 18:44:18.119103 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:18.119072    4391 generic.go:356] "Generic (PLEG): container finished" podID="5408995b-40ff-4f01-acc2-e8d8b690fc4a" containerID="bcf0984de04cdeb6d1a5c0e58e138578193744184a398ecd709138399867ff46" exitCode=0
Oct 30 18:44:18.119501 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:18.119160    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr" event={"ID":"5408995b-40ff-4f01-acc2-e8d8b690fc4a","Type":"ContainerDied","Data":"bcf0984de04cdeb6d1a5c0e58e138578193744184a398ecd709138399867ff46"}
Oct 30 18:44:18.121246 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:18.121228    4391 generic.go:356] "Generic (PLEG): container finished" podID="b0701727-b74d-4156-a436-bad574f82272" containerID="f270ad01b8452d4f3b9e50c68918a0870063a596882c808215fa01585f02f281" exitCode=0
Oct 30 18:44:18.121335 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:18.121308    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr" event={"ID":"b0701727-b74d-4156-a436-bad574f82272","Type":"ContainerDied","Data":"f270ad01b8452d4f3b9e50c68918a0870063a596882c808215fa01585f02f281"}
Oct 30 18:44:18.123305 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:18.123285    4391 generic.go:356] "Generic (PLEG): container finished" podID="12ccb2f4-40ab-491a-b839-99b7c921442c" containerID="8bd57c1f7c0edd09026d5fc56fa762a8ae8329e1576ae3d3ca8cb113c3d13079" exitCode=0
Oct 30 18:44:18.123396 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:18.123364    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9" event={"ID":"12ccb2f4-40ab-491a-b839-99b7c921442c","Type":"ContainerDied","Data":"8bd57c1f7c0edd09026d5fc56fa762a8ae8329e1576ae3d3ca8cb113c3d13079"}
Oct 30 18:44:18.125313 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:18.125295    4391 generic.go:356] "Generic (PLEG): container finished" podID="4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" containerID="f0530681e9ce19594504ca4f26a1d74bc63642b02b1af9a4c5656264105e159d" exitCode=0
Oct 30 18:44:18.125388 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:18.125340    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk" event={"ID":"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7","Type":"ContainerDied","Data":"f0530681e9ce19594504ca4f26a1d74bc63642b02b1af9a4c5656264105e159d"}
Oct 30 18:44:19.308924 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.308897    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:19.310396 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.310371    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:19.341388 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.341364    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:19.344511 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.344493    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:19.378323 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378299    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-bundle\") pod \"12ccb2f4-40ab-491a-b839-99b7c921442c\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") "
Oct 30 18:44:19.378421 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378339    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-kube-api-access-6x8c4\" (UniqueName: \"kubernetes.io/projected/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-kube-api-access-6x8c4\") pod \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") "
Oct 30 18:44:19.378421 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378356    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-util\" (UniqueName: \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-util\") pod \"b0701727-b74d-4156-a436-bad574f82272\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") "
Oct 30 18:44:19.378421 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378383    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-bundle\") pod \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") "
Oct 30 18:44:19.378421 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378403    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-bundle\") pod \"b0701727-b74d-4156-a436-bad574f82272\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") "
Oct 30 18:44:19.378641 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378438    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5408995b-40ff-4f01-acc2-e8d8b690fc4a-kube-api-access-4ztd4\" (UniqueName: \"kubernetes.io/projected/5408995b-40ff-4f01-acc2-e8d8b690fc4a-kube-api-access-4ztd4\") pod \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") "
Oct 30 18:44:19.378641 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378459    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-bundle\") pod \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") "
Oct 30 18:44:19.378641 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378496    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ccb2f4-40ab-491a-b839-99b7c921442c-kube-api-access-xkh2g\" (UniqueName: \"kubernetes.io/projected/12ccb2f4-40ab-491a-b839-99b7c921442c-kube-api-access-xkh2g\") pod \"12ccb2f4-40ab-491a-b839-99b7c921442c\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") "
Oct 30 18:44:19.379914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378979    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-bundle" pod "4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" (UID: "4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:19.379914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378988    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-bundle" pod "5408995b-40ff-4f01-acc2-e8d8b690fc4a" (UID: "5408995b-40ff-4f01-acc2-e8d8b690fc4a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:19.379914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.378939    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-util\" (UniqueName: \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-util\") pod \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\" (UID: \"5408995b-40ff-4f01-acc2-e8d8b690fc4a\") "
Oct 30 18:44:19.379914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.379150    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-util\" (UniqueName: \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-util\") pod \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\" (UID: \"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7\") "
Oct 30 18:44:19.379914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.379197    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-util\" (UniqueName: \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-util\") pod \"12ccb2f4-40ab-491a-b839-99b7c921442c\" (UID: \"12ccb2f4-40ab-491a-b839-99b7c921442c\") "
Oct 30 18:44:19.379914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.379266    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0701727-b74d-4156-a436-bad574f82272-kube-api-access-qfz8b\" (UniqueName: \"kubernetes.io/projected/b0701727-b74d-4156-a436-bad574f82272-kube-api-access-qfz8b\") pod \"b0701727-b74d-4156-a436-bad574f82272\" (UID: \"b0701727-b74d-4156-a436-bad574f82272\") "
Oct 30 18:44:19.379914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.379560    4391 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-bundle\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.379914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.379603    4391 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-bundle\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.379914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.379573    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-bundle" pod "b0701727-b74d-4156-a436-bad574f82272" (UID: "b0701727-b74d-4156-a436-bad574f82272"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:19.379914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.379654    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-bundle" pod "12ccb2f4-40ab-491a-b839-99b7c921442c" (UID: "12ccb2f4-40ab-491a-b839-99b7c921442c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:19.380993 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.380942    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-kube-api-access-6x8c4" pod "4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" (UID: "4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7"). InnerVolumeSpecName "kube-api-access-6x8c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:44:19.381193 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.381160    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5408995b-40ff-4f01-acc2-e8d8b690fc4a-kube-api-access-4ztd4" pod "5408995b-40ff-4f01-acc2-e8d8b690fc4a" (UID: "5408995b-40ff-4f01-acc2-e8d8b690fc4a"). InnerVolumeSpecName "kube-api-access-4ztd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:44:19.381623 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.381576    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ccb2f4-40ab-491a-b839-99b7c921442c-kube-api-access-xkh2g" pod "12ccb2f4-40ab-491a-b839-99b7c921442c" (UID: "12ccb2f4-40ab-491a-b839-99b7c921442c"). InnerVolumeSpecName "kube-api-access-xkh2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:44:19.382513 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.382495    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0701727-b74d-4156-a436-bad574f82272-kube-api-access-qfz8b" pod "b0701727-b74d-4156-a436-bad574f82272" (UID: "b0701727-b74d-4156-a436-bad574f82272"). InnerVolumeSpecName "kube-api-access-qfz8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:44:19.385721 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.385696    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-util" pod "5408995b-40ff-4f01-acc2-e8d8b690fc4a" (UID: "5408995b-40ff-4f01-acc2-e8d8b690fc4a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:19.387353 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.387236    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-util" pod "b0701727-b74d-4156-a436-bad574f82272" (UID: "b0701727-b74d-4156-a436-bad574f82272"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:19.388144 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.388123    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-util" pod "12ccb2f4-40ab-491a-b839-99b7c921442c" (UID: "12ccb2f4-40ab-491a-b839-99b7c921442c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:19.389441 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.389404    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-util" pod "4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" (UID: "4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:19.480234 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.480184    4391 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5408995b-40ff-4f01-acc2-e8d8b690fc4a-util\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.480234 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.480226    4391 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-util\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.480234 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.480239    4391 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-util\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.480479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.480253    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfz8b\" (UniqueName: \"kubernetes.io/projected/b0701727-b74d-4156-a436-bad574f82272-kube-api-access-qfz8b\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.480479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.480267    4391 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12ccb2f4-40ab-491a-b839-99b7c921442c-bundle\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.480479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.480279    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8c4\" (UniqueName: \"kubernetes.io/projected/4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7-kube-api-access-6x8c4\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.480479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.480290    4391 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-util\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.480479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.480301    4391 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0701727-b74d-4156-a436-bad574f82272-bundle\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.480479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.480312    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ztd4\" (UniqueName: \"kubernetes.io/projected/5408995b-40ff-4f01-acc2-e8d8b690fc4a-kube-api-access-4ztd4\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:19.480479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:19.480324    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkh2g\" (UniqueName: \"kubernetes.io/projected/12ccb2f4-40ab-491a-b839-99b7c921442c-kube-api-access-xkh2g\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:20.137643 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.137612    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9"
Oct 30 18:44:20.137643 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.137622    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4p9z9" event={"ID":"12ccb2f4-40ab-491a-b839-99b7c921442c","Type":"ContainerDied","Data":"f6bd0148dc0bb3ef0cb73ed4aeb4c04ef9ad900415335fd5d7cf46614d7139d0"}
Oct 30 18:44:20.137901 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.137692    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6bd0148dc0bb3ef0cb73ed4aeb4c04ef9ad900415335fd5d7cf46614d7139d0"
Oct 30 18:44:20.139640 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.139616    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk" event={"ID":"4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7","Type":"ContainerDied","Data":"73c42e70fcf11b88d02621a2af03f4393ae3c04852e66d80349f7b54673598ea"}
Oct 30 18:44:20.139763 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.139642    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c68gk"
Oct 30 18:44:20.139763 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.139655    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c42e70fcf11b88d02621a2af03f4393ae3c04852e66d80349f7b54673598ea"
Oct 30 18:44:20.141437 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.141414    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr"
Oct 30 18:44:20.141437 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.141420    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lx6zr" event={"ID":"5408995b-40ff-4f01-acc2-e8d8b690fc4a","Type":"ContainerDied","Data":"076c534d88a045c00dfff2877df14067424ba0e180f88f8070f795cd569b3302"}
Oct 30 18:44:20.141616 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.141457    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="076c534d88a045c00dfff2877df14067424ba0e180f88f8070f795cd569b3302"
Oct 30 18:44:20.143429 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.143408    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr" event={"ID":"b0701727-b74d-4156-a436-bad574f82272","Type":"ContainerDied","Data":"b2cabf6f29616969ba0f30aa7aea36ffdc98935d726eda8c4db011dbef01b900"}
Oct 30 18:44:20.143429 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.143420    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av7pkr"
Oct 30 18:44:20.143429 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:20.143440    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2cabf6f29616969ba0f30aa7aea36ffdc98935d726eda8c4db011dbef01b900"
Oct 30 18:44:37.997812 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.997770    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7"]
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998171    4391 cpu_manager.go:414] "containerMap: removing container" podUID="b0701727-b74d-4156-a436-bad574f82272" containerName="pull"
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998183    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0701727-b74d-4156-a436-bad574f82272" containerName="pull"
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998208    4391 cpu_manager.go:414] "containerMap: removing container" podUID="5408995b-40ff-4f01-acc2-e8d8b690fc4a" containerName="pull"
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998213    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5408995b-40ff-4f01-acc2-e8d8b690fc4a" containerName="pull"
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998221    4391 cpu_manager.go:414] "containerMap: removing container" podUID="5408995b-40ff-4f01-acc2-e8d8b690fc4a" containerName="extract"
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998226    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5408995b-40ff-4f01-acc2-e8d8b690fc4a" containerName="extract"
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998236    4391 cpu_manager.go:414] "containerMap: removing container" podUID="4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" containerName="util"
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998241    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" containerName="util"
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998249    4391 cpu_manager.go:414] "containerMap: removing container" podUID="4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" containerName="pull"
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998253    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" containerName="pull"
Oct 30 18:44:37.998253 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998258    4391 cpu_manager.go:414] "containerMap: removing container" podUID="12ccb2f4-40ab-491a-b839-99b7c921442c" containerName="pull"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998263    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ccb2f4-40ab-491a-b839-99b7c921442c" containerName="pull"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998279    4391 cpu_manager.go:414] "containerMap: removing container" podUID="4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" containerName="extract"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998283    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" containerName="extract"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998291    4391 cpu_manager.go:414] "containerMap: removing container" podUID="12ccb2f4-40ab-491a-b839-99b7c921442c" containerName="extract"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998295    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ccb2f4-40ab-491a-b839-99b7c921442c" containerName="extract"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998303    4391 cpu_manager.go:414] "containerMap: removing container" podUID="5408995b-40ff-4f01-acc2-e8d8b690fc4a" containerName="util"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998307    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5408995b-40ff-4f01-acc2-e8d8b690fc4a" containerName="util"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998312    4391 cpu_manager.go:414] "containerMap: removing container" podUID="12ccb2f4-40ab-491a-b839-99b7c921442c" containerName="util"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998316    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ccb2f4-40ab-491a-b839-99b7c921442c" containerName="util"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998322    4391 cpu_manager.go:414] "containerMap: removing container" podUID="b0701727-b74d-4156-a436-bad574f82272" containerName="extract"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998327    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0701727-b74d-4156-a436-bad574f82272" containerName="extract"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998333    4391 cpu_manager.go:414] "containerMap: removing container" podUID="b0701727-b74d-4156-a436-bad574f82272" containerName="util"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998337    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0701727-b74d-4156-a436-bad574f82272" containerName="util"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998394    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba1ad2a-abbd-4976-ad83-4f76b7bc10c7" containerName="extract"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998407    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ccb2f4-40ab-491a-b839-99b7c921442c" containerName="extract"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998416    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0701727-b74d-4156-a436-bad574f82272" containerName="extract"
Oct 30 18:44:37.998604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:37.998427    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="5408995b-40ff-4f01-acc2-e8d8b690fc4a" containerName="extract"
Oct 30 18:44:38.001610 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.001564    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7"
Oct 30 18:44:38.060193 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.060154    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c78n\" (UniqueName: \"kubernetes.io/projected/1ceaadbb-b694-4ddb-8847-6cd259db3a5e-kube-api-access-2c78n\") pod \"dns-operator-controller-manager-65bfd79598-f4lk7\" (UID: \"1ceaadbb-b694-4ddb-8847-6cd259db3a5e\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7"
Oct 30 18:44:38.103387 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.103354    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 18:44:38.103387 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.103373    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4bs6h\""
Oct 30 18:44:38.103387 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.103358    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 18:44:38.103387 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.103354    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 18:44:38.105720 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.105701    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7"]
Oct 30 18:44:38.161641 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.161598    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c78n\" (UniqueName: \"kubernetes.io/projected/1ceaadbb-b694-4ddb-8847-6cd259db3a5e-kube-api-access-2c78n\") pod \"dns-operator-controller-manager-65bfd79598-f4lk7\" (UID: \"1ceaadbb-b694-4ddb-8847-6cd259db3a5e\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7"
Oct 30 18:44:38.219948 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.219919    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c78n\" (UniqueName: \"kubernetes.io/projected/1ceaadbb-b694-4ddb-8847-6cd259db3a5e-kube-api-access-2c78n\") pod \"dns-operator-controller-manager-65bfd79598-f4lk7\" (UID: \"1ceaadbb-b694-4ddb-8847-6cd259db3a5e\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7"
Oct 30 18:44:38.314185 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.314102    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7"
Oct 30 18:44:38.680550 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:38.680513    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7"]
Oct 30 18:44:38.712080 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:44:38.712043    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ceaadbb_b694_4ddb_8847_6cd259db3a5e.slice/crio-c7d52b4b6554845fa6639e5c66b31cf6f4af93af4ca45a6608c51579d6ab6658 WatchSource:0}: Error finding container c7d52b4b6554845fa6639e5c66b31cf6f4af93af4ca45a6608c51579d6ab6658: Status 404 returned error can't find the container with id c7d52b4b6554845fa6639e5c66b31cf6f4af93af4ca45a6608c51579d6ab6658
Oct 30 18:44:39.240803 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:39.240769    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7" event={"ID":"1ceaadbb-b694-4ddb-8847-6cd259db3a5e","Type":"ContainerStarted","Data":"c7d52b4b6554845fa6639e5c66b31cf6f4af93af4ca45a6608c51579d6ab6658"}
Oct 30 18:44:41.256746 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:44:41.256702    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:44:43.267227 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:43.267186    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7" event={"ID":"1ceaadbb-b694-4ddb-8847-6cd259db3a5e","Type":"ContainerStarted","Data":"c2b77b91a8032d5b38f68f44ead1bb75e61a367297b5d6fd2d552a9ba052844c"}
Oct 30 18:44:43.267675 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:43.267295    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7"
Oct 30 18:44:43.377844 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:43.377792    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7" podStartSLOduration=2.56347812 podStartE2EDuration="6.377778608s" totalImagesPullingTime="3.814300488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:43.375680689 +0000 UTC m=+478.486413972" watchObservedRunningTime="2026-10-30 18:44:43.377778608 +0000 UTC m=+478.488511891"
Oct 30 18:44:54.275025 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:44:54.274989    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-f4lk7"
Oct 30 18:45:11.252092 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:45:11.252003    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:45:26.342234 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.342198    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"]
Oct 30 18:45:26.349026 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.349004    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.398671 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.398634    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g5j8l\""
Oct 30 18:45:26.418697 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.418669    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.418851 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.418704    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.418851 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.418733    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.418948 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.418855    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.418948 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.418905    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kcrf\" (UniqueName: \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-kube-api-access-2kcrf\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.418948 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.418924    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.419056 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.418951    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.419092 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.419059    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.419092 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.419084    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.448493 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.448463    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"]
Oct 30 18:45:26.520239 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520202    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.520422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520247    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.520422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520284    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.520422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520312    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.520422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520362    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kcrf\" (UniqueName: \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-kube-api-access-2kcrf\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.520422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520392    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.520732 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520431    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.520732 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520491    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.520732 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520521    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.520884 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520773    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.521002 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.520974    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.521068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.521006    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.521134 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.521064    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.521134 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.521120    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.522989 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.522968    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.523114 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.523092    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.579257 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.579221    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.579257 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.579255    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kcrf\" (UniqueName: \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-kube-api-access-2kcrf\") pod \"maas-default-gateway-openshift-default-75d7c785f-dszf6\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:26.662095 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:26.662069    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:27.082534 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:27.082509    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"]
Oct 30 18:45:27.084405 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:45:27.084379    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d29565c_5b84_42bb_9d1d_99c93c57ab5e.slice/crio-c295d6de63afe872b7600c02a902d49ff4c051077c690f03b58e2e093ab86391 WatchSource:0}: Error finding container c295d6de63afe872b7600c02a902d49ff4c051077c690f03b58e2e093ab86391: Status 404 returned error can't find the container with id c295d6de63afe872b7600c02a902d49ff4c051077c690f03b58e2e093ab86391
Oct 30 18:45:27.086711 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:27.086682    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:45:27.086786 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:27.086745    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:45:27.086786 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:27.086772    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:45:27.504030 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:27.503996    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" event={"ID":"1d29565c-5b84-42bb-9d1d-99c93c57ab5e","Type":"ContainerStarted","Data":"cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d"}
Oct 30 18:45:27.504434 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:27.504045    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" event={"ID":"1d29565c-5b84-42bb-9d1d-99c93c57ab5e","Type":"ContainerStarted","Data":"c295d6de63afe872b7600c02a902d49ff4c051077c690f03b58e2e093ab86391"}
Oct 30 18:45:27.616810 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:27.616755    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" podStartSLOduration=1.6167382049999999 podStartE2EDuration="1.616738205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:27.615721715 +0000 UTC m=+522.726454998" watchObservedRunningTime="2026-10-30 18:45:27.616738205 +0000 UTC m=+522.727471491"
Oct 30 18:45:27.662481 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:27.662450    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:28.667715 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:28.667684    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:29.516631 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:29.516565    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:29.518390 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:29.517549    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:45:41.252283 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:45:41.252245    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:45:50.038370 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.038337    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:45:50.041964 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.041938    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"
Oct 30 18:45:50.091893 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.091865    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 18:45:50.133754 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.133719    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hh5\" (UniqueName: \"kubernetes.io/projected/6e8e6505-6761-4a7a-bd39-b019b66d5db0-kube-api-access-s7hh5\") pod \"limitador-limitador-7c9b87d574-wrx5m\" (UID: \"6e8e6505-6761-4a7a-bd39-b019b66d5db0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"
Oct 30 18:45:50.133933 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.133865    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e8e6505-6761-4a7a-bd39-b019b66d5db0-config-file\") pod \"limitador-limitador-7c9b87d574-wrx5m\" (UID: \"6e8e6505-6761-4a7a-bd39-b019b66d5db0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"
Oct 30 18:45:50.138158 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.138133    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-szvzq\""
Oct 30 18:45:50.140412 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.140390    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:45:50.235184 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.235147    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e8e6505-6761-4a7a-bd39-b019b66d5db0-config-file\") pod \"limitador-limitador-7c9b87d574-wrx5m\" (UID: \"6e8e6505-6761-4a7a-bd39-b019b66d5db0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"
Oct 30 18:45:50.235379 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.235232    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hh5\" (UniqueName: \"kubernetes.io/projected/6e8e6505-6761-4a7a-bd39-b019b66d5db0-kube-api-access-s7hh5\") pod \"limitador-limitador-7c9b87d574-wrx5m\" (UID: \"6e8e6505-6761-4a7a-bd39-b019b66d5db0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"
Oct 30 18:45:50.235862 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.235838    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e8e6505-6761-4a7a-bd39-b019b66d5db0-config-file\") pod \"limitador-limitador-7c9b87d574-wrx5m\" (UID: \"6e8e6505-6761-4a7a-bd39-b019b66d5db0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"
Oct 30 18:45:50.291181 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.291118    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hh5\" (UniqueName: \"kubernetes.io/projected/6e8e6505-6761-4a7a-bd39-b019b66d5db0-kube-api-access-s7hh5\") pod \"limitador-limitador-7c9b87d574-wrx5m\" (UID: \"6e8e6505-6761-4a7a-bd39-b019b66d5db0\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"
Oct 30 18:45:50.353734 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.353703    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:45:50.355488 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.355460    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"
Oct 30 18:45:50.723468 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:50.723444    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:45:50.731409 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:45:50.731351    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8e6505_6761_4a7a_bd39_b019b66d5db0.slice/crio-34232fbc03977d22d19ad9d300bba9ec22e894e7c4bc61008f5d64f056ee8848 WatchSource:0}: Error finding container 34232fbc03977d22d19ad9d300bba9ec22e894e7c4bc61008f5d64f056ee8848: Status 404 returned error can't find the container with id 34232fbc03977d22d19ad9d300bba9ec22e894e7c4bc61008f5d64f056ee8848
Oct 30 18:45:51.628218 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:51.628184    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m" event={"ID":"6e8e6505-6761-4a7a-bd39-b019b66d5db0","Type":"ContainerStarted","Data":"34232fbc03977d22d19ad9d300bba9ec22e894e7c4bc61008f5d64f056ee8848"}
Oct 30 18:45:53.639921 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:53.639880    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m" event={"ID":"6e8e6505-6761-4a7a-bd39-b019b66d5db0","Type":"ContainerStarted","Data":"dfcbf52dd332c7ffd3cb6a6a57fc9bdd83c6a54e5017bd954923aab47468a038"}
Oct 30 18:45:53.640356 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:53.640044    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"
Oct 30 18:45:53.752347 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:45:53.752294    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m" podStartSLOduration=2.920521032 podStartE2EDuration="4.752279275s" totalImagesPullingTime="1.831758243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:53.749649132 +0000 UTC m=+548.860382417" watchObservedRunningTime="2026-10-30 18:45:53.752279275 +0000 UTC m=+548.863012557"
Oct 30 18:46:04.645285 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:46:04.645252    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"
Oct 30 18:46:11.252175 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:46:11.252135    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:46:41.251565 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:46:41.251528    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:47:11.253666 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:47:11.253628    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:47:27.090379 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:27.090346    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:47:35.246224 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.246190    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"]
Oct 30 18:47:35.250086 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.250063    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"
Oct 30 18:47:35.304658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.304622    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"]
Oct 30 18:47:35.314349 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.314319    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/237bc5bb-3f41-4925-8344-939672e5ed5a-cert\") pod \"llmisvc-controller-manager-6466c554d8-rv5c2\" (UID: \"237bc5bb-3f41-4925-8344-939672e5ed5a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"
Oct 30 18:47:35.314510 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.314367    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdr5q\" (UniqueName: \"kubernetes.io/projected/237bc5bb-3f41-4925-8344-939672e5ed5a-kube-api-access-zdr5q\") pod \"llmisvc-controller-manager-6466c554d8-rv5c2\" (UID: \"237bc5bb-3f41-4925-8344-939672e5ed5a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"
Oct 30 18:47:35.415977 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.415938    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/237bc5bb-3f41-4925-8344-939672e5ed5a-cert\") pod \"llmisvc-controller-manager-6466c554d8-rv5c2\" (UID: \"237bc5bb-3f41-4925-8344-939672e5ed5a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"
Oct 30 18:47:35.415977 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.415985    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdr5q\" (UniqueName: \"kubernetes.io/projected/237bc5bb-3f41-4925-8344-939672e5ed5a-kube-api-access-zdr5q\") pod \"llmisvc-controller-manager-6466c554d8-rv5c2\" (UID: \"237bc5bb-3f41-4925-8344-939672e5ed5a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"
Oct 30 18:47:35.418750 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.418720    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/237bc5bb-3f41-4925-8344-939672e5ed5a-cert\") pod \"llmisvc-controller-manager-6466c554d8-rv5c2\" (UID: \"237bc5bb-3f41-4925-8344-939672e5ed5a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"
Oct 30 18:47:35.470879 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.470849    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdr5q\" (UniqueName: \"kubernetes.io/projected/237bc5bb-3f41-4925-8344-939672e5ed5a-kube-api-access-zdr5q\") pod \"llmisvc-controller-manager-6466c554d8-rv5c2\" (UID: \"237bc5bb-3f41-4925-8344-939672e5ed5a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"
Oct 30 18:47:35.563970 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.563888    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"
Oct 30 18:47:35.945292 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.945266    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"]
Oct 30 18:47:35.947191 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:47:35.947155    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237bc5bb_3f41_4925_8344_939672e5ed5a.slice/crio-ee719b44afca86735ff9f23835df026af326156a0571d57fa38428bced876983 WatchSource:0}: Error finding container ee719b44afca86735ff9f23835df026af326156a0571d57fa38428bced876983: Status 404 returned error can't find the container with id ee719b44afca86735ff9f23835df026af326156a0571d57fa38428bced876983
Oct 30 18:47:35.948546 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:35.948530    4391 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:47:36.222944 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:36.222866    4391 generic.go:356] "Generic (PLEG): container finished" podID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerID="cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d" exitCode=0
Oct 30 18:47:36.223102 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:36.222943    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" event={"ID":"1d29565c-5b84-42bb-9d1d-99c93c57ab5e","Type":"ContainerDied","Data":"cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d"}
Oct 30 18:47:36.223328 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:36.223311    4391 scope.go:122] "RemoveContainer" containerID="cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d"
Oct 30 18:47:36.224240 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:36.224218    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2" event={"ID":"237bc5bb-3f41-4925-8344-939672e5ed5a","Type":"ContainerStarted","Data":"ee719b44afca86735ff9f23835df026af326156a0571d57fa38428bced876983"}
Oct 30 18:47:36.224335 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:36.224274    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:47:36.224382 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:36.224336    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:47:36.224382 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:36.224367    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:47:37.231270 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:37.231235    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" event={"ID":"1d29565c-5b84-42bb-9d1d-99c93c57ab5e","Type":"ContainerStarted","Data":"92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d"}
Oct 30 18:47:37.232802 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:37.232778    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2" event={"ID":"237bc5bb-3f41-4925-8344-939672e5ed5a","Type":"ContainerStarted","Data":"de4217befab85539ce198ac58d96ca202e681c413dafb7caa381c917ec39cfaa"}
Oct 30 18:47:37.232932 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:37.232853    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"
Oct 30 18:47:37.499736 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:37.499637    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2" podStartSLOduration=1.6727541970000002 podStartE2EDuration="2.499621955s" totalImagesPullingTime="826.867758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:37.498771761 +0000 UTC m=+652.609505044" watchObservedRunningTime="2026-10-30 18:47:37.499621955 +0000 UTC m=+652.610355238"
Oct 30 18:47:37.671604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:37.668042    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:47:37.671604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:37.668174    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dszf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 18:47:37.671604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:37.668217    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 18:47:38.668606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:38.666991    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dszf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 18:47:38.668606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:38.667068    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 18:47:39.665607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:39.664631    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dszf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 18:47:39.665607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:39.664700    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 18:47:40.665046 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:40.665004    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dszf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 18:47:40.665702 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:40.665669    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 18:47:41.282050 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:47:41.282011    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:47:41.669603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:41.666005    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dszf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 18:47:41.669603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:41.666069    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 18:47:42.667621 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:42.667561    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:47:42.668115 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:42.668094    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:47:42.668918 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:47:42.668899    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:48:08.240169 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:08.240088    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rv5c2"
Oct 30 18:48:08.397287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:08.397255    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"]
Oct 30 18:48:08.397500 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:08.397479    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq" podUID="fecaeae5-cc1d-40ad-a6dc-923863e99ed5" containerName="manager" containerID="cri-o://4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d" gracePeriod=30
Oct 30 18:48:08.640819 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:08.640798    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:48:08.759330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:08.759299    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-cert\" (UniqueName: \"kubernetes.io/secret/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-cert\") pod \"fecaeae5-cc1d-40ad-a6dc-923863e99ed5\" (UID: \"fecaeae5-cc1d-40ad-a6dc-923863e99ed5\") "
Oct 30 18:48:08.759521 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:08.759387    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-kube-api-access-thbdj\" (UniqueName: \"kubernetes.io/projected/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-kube-api-access-thbdj\") pod \"fecaeae5-cc1d-40ad-a6dc-923863e99ed5\" (UID: \"fecaeae5-cc1d-40ad-a6dc-923863e99ed5\") "
Oct 30 18:48:08.761708 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:08.761675    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-kube-api-access-thbdj" pod "fecaeae5-cc1d-40ad-a6dc-923863e99ed5" (UID: "fecaeae5-cc1d-40ad-a6dc-923863e99ed5"). InnerVolumeSpecName "kube-api-access-thbdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:08.761816 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:08.761684    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-cert" pod "fecaeae5-cc1d-40ad-a6dc-923863e99ed5" (UID: "fecaeae5-cc1d-40ad-a6dc-923863e99ed5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:08.860838 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:08.860802    4391 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:08.860838 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:08.860832    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thbdj\" (UniqueName: \"kubernetes.io/projected/fecaeae5-cc1d-40ad-a6dc-923863e99ed5-kube-api-access-thbdj\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:09.426926 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:09.426887    4391 generic.go:356] "Generic (PLEG): container finished" podID="fecaeae5-cc1d-40ad-a6dc-923863e99ed5" containerID="4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d" exitCode=0
Oct 30 18:48:09.427354 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:09.426931    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"
Oct 30 18:48:09.427354 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:09.426968    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq" event={"ID":"fecaeae5-cc1d-40ad-a6dc-923863e99ed5","Type":"ContainerDied","Data":"4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d"}
Oct 30 18:48:09.427354 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:09.427022    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq" event={"ID":"fecaeae5-cc1d-40ad-a6dc-923863e99ed5","Type":"ContainerDied","Data":"1984a2dc03e4b1d45c7f54d0030daadfc337f49ef565c5f2ebcc0009bc309e96"}
Oct 30 18:48:09.427354 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:09.427047    4391 scope.go:122] "RemoveContainer" containerID="4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d"
Oct 30 18:48:09.435983 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:09.435964    4391 scope.go:122] "RemoveContainer" containerID="4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d"
Oct 30 18:48:09.436243 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:48:09.436224    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d\": container with ID starting with 4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d not found: ID does not exist" containerID="4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d"
Oct 30 18:48:09.436287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:09.436254    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d"} err="failed to get container status \"4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d\": rpc error: code = NotFound desc = could not find container \"4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d\": container with ID starting with 4438160587f7a6d43d485be285b84b621bf011b8ec2e852eaeafe0a09532af1d not found: ID does not exist"
Oct 30 18:48:09.594275 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:09.594245    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"]
Oct 30 18:48:09.641000 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:09.640970    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ssfzq"]
Oct 30 18:48:11.011498 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:11.011460    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fecaeae5-cc1d-40ad-a6dc-923863e99ed5" path="/var/lib/kubelet/pods/fecaeae5-cc1d-40ad-a6dc-923863e99ed5/volumes"
Oct 30 18:48:11.251898 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:48:11.251862    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:48:32.188489 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:32.188444    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:48:39.912160 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:39.912124    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:48:41.252907 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:48:41.252868    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:48:48.056859 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:48.056824    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:48:57.758443 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:48:57.758402    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:49:11.252181 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:49:11.252141    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:49:13.222783 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.222739    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"]
Oct 30 18:49:13.223347 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.223332    4391 cpu_manager.go:414] "containerMap: removing container" podUID="fecaeae5-cc1d-40ad-a6dc-923863e99ed5" containerName="manager"
Oct 30 18:49:13.223390 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.223348    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecaeae5-cc1d-40ad-a6dc-923863e99ed5" containerName="manager"
Oct 30 18:49:13.223488 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.223476    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecaeae5-cc1d-40ad-a6dc-923863e99ed5" containerName="manager"
Oct 30 18:49:13.226181 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.226164    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.261619 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.261567    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.261769 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.261640    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.261769 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.261671    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6090f1-2038-4579-9627-a1309e77367b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.261848 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.261754    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntmd2\" (UniqueName: \"kubernetes.io/projected/ef6090f1-2038-4579-9627-a1309e77367b-kube-api-access-ntmd2\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.261883 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.261840    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.261883 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.261865    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.261944 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.261888    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6090f1-2038-4579-9627-a1309e77367b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.261996 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.261982    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.262035 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.262016    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6090f1-2038-4579-9627-a1309e77367b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.324667 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.324631    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"]
Oct 30 18:49:13.362696 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.362662    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.362945 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.362707    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.362945 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.362726    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6090f1-2038-4579-9627-a1309e77367b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.362945 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.362762    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntmd2\" (UniqueName: \"kubernetes.io/projected/ef6090f1-2038-4579-9627-a1309e77367b-kube-api-access-ntmd2\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.362945 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.362819    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.362945 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.362842    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.362945 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.362874    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6090f1-2038-4579-9627-a1309e77367b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.362945 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.362921    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.363308 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.362959    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6090f1-2038-4579-9627-a1309e77367b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.363308 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.363194    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.363308 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.363271    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.363445 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.363403    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.363714 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.363696    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6090f1-2038-4579-9627-a1309e77367b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.363828 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.363715    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.365071 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.365051    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6090f1-2038-4579-9627-a1309e77367b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.365281 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.365265    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6090f1-2038-4579-9627-a1309e77367b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.416904 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.416867    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntmd2\" (UniqueName: \"kubernetes.io/projected/ef6090f1-2038-4579-9627-a1309e77367b-kube-api-access-ntmd2\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.416904 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.416897    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6090f1-2038-4579-9627-a1309e77367b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-gpdv4\" (UID: \"ef6090f1-2038-4579-9627-a1309e77367b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.540108 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.540020    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:13.965465 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.965438    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"]
Oct 30 18:49:13.968003 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:49:13.967972    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6090f1_2038_4579_9627_a1309e77367b.slice/crio-b1358809ddaac891633442ba78e0041bb83073bd495b28f854e746720faa5abc WatchSource:0}: Error finding container b1358809ddaac891633442ba78e0041bb83073bd495b28f854e746720faa5abc: Status 404 returned error can't find the container with id b1358809ddaac891633442ba78e0041bb83073bd495b28f854e746720faa5abc
Oct 30 18:49:13.970183 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.970149    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:49:13.970263 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.970215    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:49:13.970263 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:13.970246    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:49:14.802458 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:14.802423    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4" event={"ID":"ef6090f1-2038-4579-9627-a1309e77367b","Type":"ContainerStarted","Data":"3381d75d534d9d71542e4973b91b8358a2089d102989aa52a395ef93de31bf05"}
Oct 30 18:49:14.802879 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:14.802496    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4" event={"ID":"ef6090f1-2038-4579-9627-a1309e77367b","Type":"ContainerStarted","Data":"b1358809ddaac891633442ba78e0041bb83073bd495b28f854e746720faa5abc"}
Oct 30 18:49:14.967603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:14.962651    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4" podStartSLOduration=1.9626305080000002 podStartE2EDuration="1.962630508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:14.961079626 +0000 UTC m=+750.071812913" watchObservedRunningTime="2026-10-30 18:49:14.962630508 +0000 UTC m=+750.073363791"
Oct 30 18:49:15.540251 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:15.540215    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:15.541269 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:15.541249    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gpdv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:49:15.541333 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:15.541312    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4" podUID="ef6090f1-2038-4579-9627-a1309e77367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:49:16.541423 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:16.541384    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gpdv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:49:16.541892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:16.541457    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4" podUID="ef6090f1-2038-4579-9627-a1309e77367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:49:17.540400 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:17.540358    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gpdv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:49:17.540565 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:17.540429    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4" podUID="ef6090f1-2038-4579-9627-a1309e77367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:49:18.540771 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:18.540732    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gpdv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:49:18.541173 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:18.540795    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4" podUID="ef6090f1-2038-4579-9627-a1309e77367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:49:19.540645 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:19.540608    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gpdv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:49:19.540818 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:19.540674    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4" podUID="ef6090f1-2038-4579-9627-a1309e77367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:49:20.541166 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:20.541126    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gpdv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:49:20.541539 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:20.541206    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4" podUID="ef6090f1-2038-4579-9627-a1309e77367b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 18:49:22.258114 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:22.258078    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:22.258773 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:22.258744    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:22.260214 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:49:22.259780    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gpdv4"
Oct 30 18:49:41.251935 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:49:41.251899    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:50:11.252343 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:50:11.252303    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:50:40.475308 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.475267    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"]
Oct 30 18:50:40.483294 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.483262    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 18:50:40.502281 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.502253    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05e5c49-b96a-432f-8a57-669cde74121d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x\" (UID: \"c05e5c49-b96a-432f-8a57-669cde74121d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 18:50:40.502457 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.502370    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdb7h\" (UniqueName: \"kubernetes.io/projected/c05e5c49-b96a-432f-8a57-669cde74121d-kube-api-access-zdb7h\") pod \"payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x\" (UID: \"c05e5c49-b96a-432f-8a57-669cde74121d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 18:50:40.542256 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.541869    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"]
Oct 30 18:50:40.543883 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.543852    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f5705f-dockercfg-qhwqf\""
Oct 30 18:50:40.544036 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.543910    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f5705f\""
Oct 30 18:50:40.592084 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.592050    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:50:40.602919 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.602887    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05e5c49-b96a-432f-8a57-669cde74121d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x\" (UID: \"c05e5c49-b96a-432f-8a57-669cde74121d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 18:50:40.603096 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.602975    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdb7h\" (UniqueName: \"kubernetes.io/projected/c05e5c49-b96a-432f-8a57-669cde74121d-kube-api-access-zdb7h\") pod \"payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x\" (UID: \"c05e5c49-b96a-432f-8a57-669cde74121d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 18:50:40.603699 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.603675    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05e5c49-b96a-432f-8a57-669cde74121d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x\" (UID: \"c05e5c49-b96a-432f-8a57-669cde74121d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 18:50:40.663023 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.662990    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdb7h\" (UniqueName: \"kubernetes.io/projected/c05e5c49-b96a-432f-8a57-669cde74121d-kube-api-access-zdb7h\") pod \"payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x\" (UID: \"c05e5c49-b96a-432f-8a57-669cde74121d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 18:50:40.799490 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:40.799407    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 18:50:41.167665 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:41.167631    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"]
Oct 30 18:50:41.169255 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:50:41.169229    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05e5c49_b96a_432f_8a57_669cde74121d.slice/crio-b4c4acb7c97c961bd0e1441343ca305c577cb59a2dadeb79bad9cb93b13dd14e WatchSource:0}: Error finding container b4c4acb7c97c961bd0e1441343ca305c577cb59a2dadeb79bad9cb93b13dd14e: Status 404 returned error can't find the container with id b4c4acb7c97c961bd0e1441343ca305c577cb59a2dadeb79bad9cb93b13dd14e
Oct 30 18:50:41.265089 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:50:41.265050    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:50:41.269970 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:41.269938    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x" event={"ID":"c05e5c49-b96a-432f-8a57-669cde74121d","Type":"ContainerStarted","Data":"b4c4acb7c97c961bd0e1441343ca305c577cb59a2dadeb79bad9cb93b13dd14e"}
Oct 30 18:50:42.808786 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.808754    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"]
Oct 30 18:50:42.812009 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.811992    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 18:50:42.830851 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.830820    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltww\" (UniqueName: \"kubernetes.io/projected/71648928-19f6-4214-b199-9b336e6bde04-kube-api-access-7ltww\") pod \"payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5\" (UID: \"71648928-19f6-4214-b199-9b336e6bde04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 18:50:42.831045 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.830975    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71648928-19f6-4214-b199-9b336e6bde04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5\" (UID: \"71648928-19f6-4214-b199-9b336e6bde04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 18:50:42.867913 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.867882    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"]
Oct 30 18:50:42.910132 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.909965    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a1617e-dockercfg-7qg7z\""
Oct 30 18:50:42.910132 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.910061    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a1617e\""
Oct 30 18:50:42.932365 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.932334    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71648928-19f6-4214-b199-9b336e6bde04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5\" (UID: \"71648928-19f6-4214-b199-9b336e6bde04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 18:50:42.932524 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.932392    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltww\" (UniqueName: \"kubernetes.io/projected/71648928-19f6-4214-b199-9b336e6bde04-kube-api-access-7ltww\") pod \"payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5\" (UID: \"71648928-19f6-4214-b199-9b336e6bde04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 18:50:42.933303 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.933278    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71648928-19f6-4214-b199-9b336e6bde04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5\" (UID: \"71648928-19f6-4214-b199-9b336e6bde04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 18:50:42.989914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:42.989877    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltww\" (UniqueName: \"kubernetes.io/projected/71648928-19f6-4214-b199-9b336e6bde04-kube-api-access-7ltww\") pod \"payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5\" (UID: \"71648928-19f6-4214-b199-9b336e6bde04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 18:50:43.126277 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:43.126196    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 18:50:43.504138 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:43.504100    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"]
Oct 30 18:50:43.514643 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:50:43.510324    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71648928_19f6_4214_b199_9b336e6bde04.slice/crio-fb425742fdf5ad76cf5342cf1b08c1e3fcf92a239a24724f4541fcc74828803a WatchSource:0}: Error finding container fb425742fdf5ad76cf5342cf1b08c1e3fcf92a239a24724f4541fcc74828803a: Status 404 returned error can't find the container with id fb425742fdf5ad76cf5342cf1b08c1e3fcf92a239a24724f4541fcc74828803a
Oct 30 18:50:44.289022 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:44.288991    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5" event={"ID":"71648928-19f6-4214-b199-9b336e6bde04","Type":"ContainerStarted","Data":"fb425742fdf5ad76cf5342cf1b08c1e3fcf92a239a24724f4541fcc74828803a"}
Oct 30 18:50:44.290586 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:44.290561    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x" event={"ID":"c05e5c49-b96a-432f-8a57-669cde74121d","Type":"ContainerStarted","Data":"f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495"}
Oct 30 18:50:44.290709 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:44.290635    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 18:50:44.496655 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:44.496603    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x" podStartSLOduration=1.794175329 podStartE2EDuration="4.496576546s" totalImagesPullingTime="2.702401217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:44.493906536 +0000 UTC m=+839.604639819" watchObservedRunningTime="2026-10-30 18:50:44.496576546 +0000 UTC m=+839.607309829"
Oct 30 18:50:45.297470 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:45.297432    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5" event={"ID":"71648928-19f6-4214-b199-9b336e6bde04","Type":"ContainerStarted","Data":"828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21"}
Oct 30 18:50:45.298002 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:45.297602    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 18:50:45.408919 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:45.408865    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5" podStartSLOduration=2.559234753 podStartE2EDuration="3.408848261s" totalImagesPullingTime="849.613508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:45.406687735 +0000 UTC m=+840.517421019" watchObservedRunningTime="2026-10-30 18:50:45.408848261 +0000 UTC m=+840.519581544"
Oct 30 18:50:49.170649 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:49.170601    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:50:55.300933 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:55.300900    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 18:50:56.306403 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:56.306369    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 18:50:57.038432 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:50:57.038399    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:51:03.782099 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.782060    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"]
Oct 30 18:51:03.784957 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.784936    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:03.881265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.881229    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-75e91b-serving-cert\""
Oct 30 18:51:03.881265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.881251    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:51:03.881482 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.881258    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:51:03.881482 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.881288    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 18:51:03.881566 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.881554    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-977zh\""
Oct 30 18:51:03.884033 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.884014    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"]
Oct 30 18:51:03.950932 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.950900    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f58h\" (UniqueName: \"kubernetes.io/projected/97483113-cab0-47f1-9ded-0f2814455361-kube-api-access-8f58h\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:03.951102 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.950968    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-metrics-certs\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:03.951147 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.951095    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-maas-api-tls\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:03.951147 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:03.951140    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97483113-cab0-47f1-9ded-0f2814455361-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:04.053747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:04.053634    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f58h\" (UniqueName: \"kubernetes.io/projected/97483113-cab0-47f1-9ded-0f2814455361-kube-api-access-8f58h\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:04.053747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:04.053722    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-metrics-certs\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:04.053971 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:04.053814    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-maas-api-tls\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:04.053971 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:04.053861    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97483113-cab0-47f1-9ded-0f2814455361-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:04.054745 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:51:04.054720    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-serving-cert: secret "maas-api-e2e-worker-w4-75e91b-serving-cert" not found
Oct 30 18:51:04.058377 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:51:04.055022    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-maas-api-tls podName:97483113-cab0-47f1-9ded-0f2814455361 nodeName:}" failed. No retries permitted until 2026-10-30 18:51:04.554994832 +0000 UTC m=+859.665728118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-maas-api-tls") pod "maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc" (UID: "97483113-cab0-47f1-9ded-0f2814455361") : secret "maas-api-e2e-worker-w4-75e91b-serving-cert" not found
Oct 30 18:51:04.060189 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:04.060001    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97483113-cab0-47f1-9ded-0f2814455361-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:04.068001 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:04.067975    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-metrics-certs\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:04.110190 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:04.110160    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f58h\" (UniqueName: \"kubernetes.io/projected/97483113-cab0-47f1-9ded-0f2814455361-kube-api-access-8f58h\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:04.559058 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:04.559020    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-maas-api-tls\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:04.561680 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:04.561649    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-maas-api-tls\") pod \"maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:04.696812 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:04.696775    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:05.070697 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:05.070667    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"]
Oct 30 18:51:05.073304 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:51:05.073280    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97483113_cab0_47f1_9ded_0f2814455361.slice/crio-d27a51e8ec7e9fedd7c535a5bff00e5e893ebc4f64ba011bee8ba77582f1c17e WatchSource:0}: Error finding container d27a51e8ec7e9fedd7c535a5bff00e5e893ebc4f64ba011bee8ba77582f1c17e: Status 404 returned error can't find the container with id d27a51e8ec7e9fedd7c535a5bff00e5e893ebc4f64ba011bee8ba77582f1c17e
Oct 30 18:51:05.410343 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:05.410306    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc" event={"ID":"97483113-cab0-47f1-9ded-0f2814455361","Type":"ContainerStarted","Data":"d27a51e8ec7e9fedd7c535a5bff00e5e893ebc4f64ba011bee8ba77582f1c17e"}
Oct 30 18:51:08.430398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:08.430366    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc" event={"ID":"97483113-cab0-47f1-9ded-0f2814455361","Type":"ContainerStarted","Data":"99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21"}
Oct 30 18:51:08.430870 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:08.430459    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:08.542686 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:08.542630    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc" podStartSLOduration=2.467740828 podStartE2EDuration="5.542617047s" totalImagesPullingTime="3.074876219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:08.541327157 +0000 UTC m=+863.652060441" watchObservedRunningTime="2026-10-30 18:51:08.542617047 +0000 UTC m=+863.653350363"
Oct 30 18:51:11.252117 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:51:11.252074    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:51:13.549515 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.549477    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"]
Oct 30 18:51:13.552738 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.552714    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.602885 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.602852    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-a1617e-serving-cert\""
Oct 30 18:51:13.650699 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.650656    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-metrics-certs\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.650858 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.650706    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.650858 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.650734    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v569v\" (UniqueName: \"kubernetes.io/projected/b59a45ca-edd6-45e3-b55b-1a4456ab0701-kube-api-access-v569v\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.650990 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.650859    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59a45ca-edd6-45e3-b55b-1a4456ab0701-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.651842 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.651820    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"]
Oct 30 18:51:13.751498 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.751465    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-metrics-certs\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.751677 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.751519    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.751677 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:51:13.751636    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-serving-cert: secret "maas-api-e2e-worker-w3-a1617e-serving-cert" not found
Oct 30 18:51:13.751677 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.751651    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v569v\" (UniqueName: \"kubernetes.io/projected/b59a45ca-edd6-45e3-b55b-1a4456ab0701-kube-api-access-v569v\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.751866 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:51:13.751703    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-maas-api-tls podName:b59a45ca-edd6-45e3-b55b-1a4456ab0701 nodeName:}" failed. No retries permitted until 2026-10-30 18:51:14.251680503 +0000 UTC m=+869.362413780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-maas-api-tls") pod "maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf" (UID: "b59a45ca-edd6-45e3-b55b-1a4456ab0701") : secret "maas-api-e2e-worker-w3-a1617e-serving-cert" not found
Oct 30 18:51:13.751866 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.751769    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59a45ca-edd6-45e3-b55b-1a4456ab0701-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.752368 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.752344    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59a45ca-edd6-45e3-b55b-1a4456ab0701-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.753930 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.753911    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-metrics-certs\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:13.806723 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:13.806646    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v569v\" (UniqueName: \"kubernetes.io/projected/b59a45ca-edd6-45e3-b55b-1a4456ab0701-kube-api-access-v569v\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:14.256508 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:14.256471    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:14.258964 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:14.258933    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:14.441558 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:14.441529    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 18:51:14.465144 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:14.465117    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:14.835108 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:14.835083    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"]
Oct 30 18:51:14.887795 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:14.887757    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:51:15.468232 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:15.468198    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf" event={"ID":"b59a45ca-edd6-45e3-b55b-1a4456ab0701","Type":"ContainerStarted","Data":"fa41dbaf254f4cbbdbd181cf8f574317786869c775f7ff291dec22471ba2b7fb"}
Oct 30 18:51:16.475457 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:16.475421    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf" event={"ID":"b59a45ca-edd6-45e3-b55b-1a4456ab0701","Type":"ContainerStarted","Data":"4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02"}
Oct 30 18:51:16.475872 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:16.475485    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:16.585820 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:16.585770    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf" podStartSLOduration=2.747867738 podStartE2EDuration="3.585754516s" totalImagesPullingTime="837.886778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:16.584331483 +0000 UTC m=+871.695064766" watchObservedRunningTime="2026-10-30 18:51:16.585754516 +0000 UTC m=+871.696487800"
Oct 30 18:51:22.486248 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:22.486216    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 18:51:31.966985 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:31.966939    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:51:41.252200 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:51:41.252163    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:51:50.605407 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:51:50.605371    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:52:08.969454 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:08.969420    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:52:11.251623 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:52:11.251564    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:52:22.964249 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:22.964212    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"]
Oct 30 18:52:22.968567 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:22.968544    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.020789 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.020754    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"]
Oct 30 18:52:23.064862 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.064828    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-649d16\"/\"openshift-service-ca.crt\""
Oct 30 18:52:23.065042 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.064835    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-649d16\"/\"kube-root-ca.crt\""
Oct 30 18:52:23.067499 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.067478    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-649d16\"/\"simulator-w6-649d16-kserve-self-signed-certs\""
Oct 30 18:52:23.067548 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.067523    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-649d16\"/\"default-dockercfg-m5khk\""
Oct 30 18:52:23.082010 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.081982    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9spb\" (UniqueName: \"kubernetes.io/projected/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-kube-api-access-m9spb\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.082154 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.082034    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-model-cache\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.082154 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.082118    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-home\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.082224 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.082154    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-dshm\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.082258 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.082238    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tls-certs\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.082290 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.082263    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tmp-dir\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.183074 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.183036    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-model-cache\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.183074 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.183079    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-home\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.183331 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.183101    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-dshm\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.183331 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.183146    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tls-certs\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.183331 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.183176    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tmp-dir\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.183331 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.183257    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9spb\" (UniqueName: \"kubernetes.io/projected/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-kube-api-access-m9spb\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.183547 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.183478    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-home\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.183622 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.183560    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-model-cache\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.183833 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.183808    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tmp-dir\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.185323 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.185296    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-dshm\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.185571 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.185554    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tls-certs\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.238171 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.238094    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9spb\" (UniqueName: \"kubernetes.io/projected/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-kube-api-access-m9spb\") pod \"simulator-w6-649d16-kserve-7fd995d99c-m6vgx\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") " pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.280501 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.280470    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:23.654712 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.654688    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"]
Oct 30 18:52:23.656704 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:52:23.656670    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bc53cc7_83c4_495b_8176_a15ef5ce4ecf.slice/crio-15bb42fc022ef4614d1cd54bc377f497490d70b71a32f42c511d1601518a0b6d WatchSource:0}: Error finding container 15bb42fc022ef4614d1cd54bc377f497490d70b71a32f42c511d1601518a0b6d: Status 404 returned error can't find the container with id 15bb42fc022ef4614d1cd54bc377f497490d70b71a32f42c511d1601518a0b6d
Oct 30 18:52:23.869298 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:23.869260    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" event={"ID":"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf","Type":"ContainerStarted","Data":"15bb42fc022ef4614d1cd54bc377f497490d70b71a32f42c511d1601518a0b6d"}
Oct 30 18:52:25.883309 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:25.883273    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" event={"ID":"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf","Type":"ContainerStarted","Data":"0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346"}
Oct 30 18:52:25.993693 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:25.993640    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podStartSLOduration=2.414330101 podStartE2EDuration="3.993627086s" totalImagesPullingTime="1.579296985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:25.99113025 +0000 UTC m=+941.101863529" watchObservedRunningTime="2026-10-30 18:52:25.993627086 +0000 UTC m=+941.104360369"
Oct 30 18:52:26.281136 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:26.281102    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:28.166857 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:28.166774    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:52:33.281421 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:33.281382    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:33.294922 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:33.294891    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:33.938826 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:33.938796    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:52:41.253287 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:52:41.253249    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:52:48.017411 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:48.017374    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:52:56.164831 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.164796    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"]
Oct 30 18:52:56.168756 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.168736    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.221974 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.221943    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"]
Oct 30 18:52:56.253309 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.253277    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ec8dab\"/\"openshift-service-ca.crt\""
Oct 30 18:52:56.253309 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.253301    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ec8dab\"/\"kube-root-ca.crt\""
Oct 30 18:52:56.265979 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.265952    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec8dab\"/\"premium-w2-ec8dab-kserve-self-signed-certs\""
Oct 30 18:52:56.265979 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.265967    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec8dab\"/\"default-dockercfg-kvwpm\""
Oct 30 18:52:56.301382 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.301340    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqdx\" (UniqueName: \"kubernetes.io/projected/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-kube-api-access-ldqdx\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.301382 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.301384    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tmp-dir\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.301650 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.301405    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-home\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.301650 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.301556    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tls-certs\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.301774 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.301659    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-dshm\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.301774 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.301698    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-model-cache\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.402470 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.402432    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqdx\" (UniqueName: \"kubernetes.io/projected/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-kube-api-access-ldqdx\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.402673 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.402475    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tmp-dir\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.402673 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.402497    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-home\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.402673 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.402551    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tls-certs\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.402673 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.402648    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-dshm\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.402959 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.402697    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-model-cache\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.402959 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.402947    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tmp-dir\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.403055 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.403029    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-home\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.403091 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.403070    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-model-cache\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.405089 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.405060    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-dshm\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.405328 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.405310    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tls-certs\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.456820 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.456749    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqdx\" (UniqueName: \"kubernetes.io/projected/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-kube-api-access-ldqdx\") pod \"premium-w2-ec8dab-kserve-6db555c58f-dwbpf\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") " pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.480619 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.480571    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:52:56.874712 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:56.874688    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"]
Oct 30 18:52:56.876607 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:52:56.876552    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd19633f_bc62_4fc1_9a33_f0e40a541c8b.slice/crio-7d961d411cb670c2ee655027e2fc7acad43f493cab564a9957b183bd98862a3e WatchSource:0}: Error finding container 7d961d411cb670c2ee655027e2fc7acad43f493cab564a9957b183bd98862a3e: Status 404 returned error can't find the container with id 7d961d411cb670c2ee655027e2fc7acad43f493cab564a9957b183bd98862a3e
Oct 30 18:52:57.050985 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:57.050951    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" event={"ID":"dd19633f-bc62-4fc1-9a33-f0e40a541c8b","Type":"ContainerStarted","Data":"649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3"}
Oct 30 18:52:57.051176 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:57.050996    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" event={"ID":"dd19633f-bc62-4fc1-9a33-f0e40a541c8b","Type":"ContainerStarted","Data":"7d961d411cb670c2ee655027e2fc7acad43f493cab564a9957b183bd98862a3e"}
Oct 30 18:52:57.159031 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:57.158979    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podStartSLOduration=1.158965853 podStartE2EDuration="1.158965853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:57.157726526 +0000 UTC m=+972.268459811" watchObservedRunningTime="2026-10-30 18:52:57.158965853 +0000 UTC m=+972.269699136"
Oct 30 18:52:57.481465 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:52:57.481396    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:53:06.481463 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:06.481426    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:53:06.493370 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:06.493337    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:53:06.494115 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:06.494090    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:53:07.116621 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:07.116588    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:53:11.252552 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:53:11.252514    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:53:30.490976 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:30.490945    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:53:41.253422 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:53:41.253379    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:53:42.680786 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.680752    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"]
Oct 30 18:53:42.684884 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.684858    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.770950 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.770914    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-dshm\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.771135 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.770967    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a69d375-f817-4870-951d-e6969a279191-tls-certs\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.771135 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.771078    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-home\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.771243 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.771139    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-tmp-dir\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.771281 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.771238    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-model-cache\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.771281 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.771264    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5w7c\" (UniqueName: \"kubernetes.io/projected/0a69d375-f817-4870-951d-e6969a279191-kube-api-access-n5w7c\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.783296 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.783269    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-649d16\"/\"premium-w6-649d16-kserve-self-signed-certs\""
Oct 30 18:53:42.785846 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.785826    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"]
Oct 30 18:53:42.872520 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.872486    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-dshm\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.872717 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.872533    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a69d375-f817-4870-951d-e6969a279191-tls-certs\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.872717 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.872563    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-home\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.872717 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.872626    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-tmp-dir\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.872717 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.872689    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-model-cache\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.872717 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.872714    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5w7c\" (UniqueName: \"kubernetes.io/projected/0a69d375-f817-4870-951d-e6969a279191-kube-api-access-n5w7c\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.873015 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.872988    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-home\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.873107 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.873086    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-model-cache\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.873107 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.873097    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-tmp-dir\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.874936 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.874913    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-dshm\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.875066 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.875047    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a69d375-f817-4870-951d-e6969a279191-tls-certs\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.927804 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.927770    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5w7c\" (UniqueName: \"kubernetes.io/projected/0a69d375-f817-4870-951d-e6969a279191-kube-api-access-n5w7c\") pod \"premium-w6-649d16-kserve-8b469dd4d-xvl5q\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") " pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:42.997478 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:42.997393    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:43.413367 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:43.413337    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"]
Oct 30 18:53:43.415337 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:53:43.415311    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a69d375_f817_4870_951d_e6969a279191.slice/crio-5c093e525054c5bcfe7de683e57d03c62f2e180db51cf5d996eb3848ec7437fc WatchSource:0}: Error finding container 5c093e525054c5bcfe7de683e57d03c62f2e180db51cf5d996eb3848ec7437fc: Status 404 returned error can't find the container with id 5c093e525054c5bcfe7de683e57d03c62f2e180db51cf5d996eb3848ec7437fc
Oct 30 18:53:44.312916 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:44.312877    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" event={"ID":"0a69d375-f817-4870-951d-e6969a279191","Type":"ContainerStarted","Data":"ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef"}
Oct 30 18:53:44.312916 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:44.312918    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" event={"ID":"0a69d375-f817-4870-951d-e6969a279191","Type":"ContainerStarted","Data":"5c093e525054c5bcfe7de683e57d03c62f2e180db51cf5d996eb3848ec7437fc"}
Oct 30 18:53:44.422171 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:44.422118    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podStartSLOduration=2.422104321 podStartE2EDuration="2.422104321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:44.420638974 +0000 UTC m=+1019.531372259" watchObservedRunningTime="2026-10-30 18:53:44.422104321 +0000 UTC m=+1019.532837604"
Oct 30 18:53:44.997805 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:44.997776    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:52.997801 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:52.997757    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:53.012082 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:53.012046    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:53.379979 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:53.379898    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:53:55.645450 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:53:55.645416    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:54:01.826708 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.826619    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"]
Oct 30 18:54:01.831022 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.830996    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.872040 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.871806    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-model-cache\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.872040 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.871970    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7db\" (UniqueName: \"kubernetes.io/projected/ff3d235d-adec-4667-9553-3fe55500a0a8-kube-api-access-rc7db\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.872040 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.872005    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-dshm\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.872040 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.872043    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-home\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.872384 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.872070    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3d235d-adec-4667-9553-3fe55500a0a8-tls-certs\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.872384 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.872114    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-tmp-dir\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.927772 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.927733    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-75e91b\"/\"kube-root-ca.crt\""
Oct 30 18:54:01.927772 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.927743    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-75e91b\"/\"openshift-service-ca.crt\""
Oct 30 18:54:01.927996 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.927799    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-75e91b\"/\"default-dockercfg-qrgx8\""
Oct 30 18:54:01.928053 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.928030    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-75e91b\"/\"premium-w4-75e91b-kserve-self-signed-certs\""
Oct 30 18:54:01.930571 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.930546    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"]
Oct 30 18:54:01.989456 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.989410    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7db\" (UniqueName: \"kubernetes.io/projected/ff3d235d-adec-4667-9553-3fe55500a0a8-kube-api-access-rc7db\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.989725 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.989469    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-dshm\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.989725 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.989549    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-home\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.989725 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.989574    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3d235d-adec-4667-9553-3fe55500a0a8-tls-certs\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.989725 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.989631    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-tmp-dir\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.989949 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.989768    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-model-cache\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.990319 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.990295    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-model-cache\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.990969 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.990943    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-home\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.991976 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.991950    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-tmp-dir\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:01.994832 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:01.994736    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-dshm\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:02.002504 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:02.002471    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3d235d-adec-4667-9553-3fe55500a0a8-tls-certs\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:02.045780 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:02.045744    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7db\" (UniqueName: \"kubernetes.io/projected/ff3d235d-adec-4667-9553-3fe55500a0a8-kube-api-access-rc7db\") pod \"premium-w4-75e91b-kserve-6b9958f8fc-h9s4h\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") " pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:02.143713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:02.143615    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:02.543635 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:02.543589    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"]
Oct 30 18:54:02.545763 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:54:02.545735    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff3d235d_adec_4667_9553_3fe55500a0a8.slice/crio-ee794ea3a46b23d16fa234ea57827ca2ad3d56d8e889c1bc2f1ea1aace3a7412 WatchSource:0}: Error finding container ee794ea3a46b23d16fa234ea57827ca2ad3d56d8e889c1bc2f1ea1aace3a7412: Status 404 returned error can't find the container with id ee794ea3a46b23d16fa234ea57827ca2ad3d56d8e889c1bc2f1ea1aace3a7412
Oct 30 18:54:03.426459 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:03.426416    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" event={"ID":"ff3d235d-adec-4667-9553-3fe55500a0a8","Type":"ContainerStarted","Data":"abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf"}
Oct 30 18:54:03.426892 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:03.426471    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" event={"ID":"ff3d235d-adec-4667-9553-3fe55500a0a8","Type":"ContainerStarted","Data":"ee794ea3a46b23d16fa234ea57827ca2ad3d56d8e889c1bc2f1ea1aace3a7412"}
Oct 30 18:54:03.536874 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:03.536817    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podStartSLOduration=2.536803928 podStartE2EDuration="2.536803928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:03.533990784 +0000 UTC m=+1038.644724066" watchObservedRunningTime="2026-10-30 18:54:03.536803928 +0000 UTC m=+1038.647537211"
Oct 30 18:54:04.144055 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:04.144017    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:11.252719 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:54:11.252681    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:54:12.143777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:12.143744    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:12.156265 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:12.156241    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:12.504675 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:12.504647    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:54:14.096749 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.096715    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"]
Oct 30 18:54:14.100688 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.100666    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.197446 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.197419    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a1617e\"/\"kube-root-ca.crt\""
Oct 30 18:54:14.197636 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.197485    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a1617e\"/\"openshift-service-ca.crt\""
Oct 30 18:54:14.197636 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.197524    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a1617e\"/\"premium-w3-a1617e-kserve-self-signed-certs\""
Oct 30 18:54:14.197713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.197674    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a1617e\"/\"default-dockercfg-xbgmq\""
Oct 30 18:54:14.200239 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.200216    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"]
Oct 30 18:54:14.221355 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.221333    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-home\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.221476 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.221361    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-model-cache\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.221476 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.221385    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tls-certs\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.221476 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.221451    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tmp-dir\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.221576 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.221490    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxd9\" (UniqueName: \"kubernetes.io/projected/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-kube-api-access-bmxd9\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.221576 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.221534    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-dshm\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.322290 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.322254    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-home\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.322479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.322293    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-model-cache\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.322479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.322420    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tls-certs\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.322479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.322471    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tmp-dir\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.322676 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.322501    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxd9\" (UniqueName: \"kubernetes.io/projected/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-kube-api-access-bmxd9\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.322676 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.322551    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-dshm\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.322782 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.322727    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-model-cache\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.322823 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.322784    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-home\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.323174 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.323147    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tmp-dir\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.324792 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.324766    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-dshm\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.325040 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.325021    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tls-certs\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.381639 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.381550    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxd9\" (UniqueName: \"kubernetes.io/projected/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-kube-api-access-bmxd9\") pod \"premium-w3-a1617e-kserve-6695fcbc9d-xcvn6\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") " pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.413843 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.413816    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:14.828196 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:14.828169    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"]
Oct 30 18:54:14.831259 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:54:14.831224    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod065fbb95_3d5b_47fb_8daa_bc9cc33a5b10.slice/crio-50ec1c88352be87f4171ec44216d2b131e38a5c6ddeb23e3e762e64a86c82faa WatchSource:0}: Error finding container 50ec1c88352be87f4171ec44216d2b131e38a5c6ddeb23e3e762e64a86c82faa: Status 404 returned error can't find the container with id 50ec1c88352be87f4171ec44216d2b131e38a5c6ddeb23e3e762e64a86c82faa
Oct 30 18:54:15.518522 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:15.518486    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" event={"ID":"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10","Type":"ContainerStarted","Data":"4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f"}
Oct 30 18:54:15.518522 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:15.518531    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" event={"ID":"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10","Type":"ContainerStarted","Data":"50ec1c88352be87f4171ec44216d2b131e38a5c6ddeb23e3e762e64a86c82faa"}
Oct 30 18:54:15.627256 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:15.627206    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podStartSLOduration=1.627193207 podStartE2EDuration="1.627193207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:15.626380662 +0000 UTC m=+1050.737113944" watchObservedRunningTime="2026-10-30 18:54:15.627193207 +0000 UTC m=+1050.737926489"
Oct 30 18:54:16.414057 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:16.414022    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:23.217547 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:23.217515    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:54:24.270785 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.270749    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"]
Oct 30 18:54:24.274944 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.274923    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.372998 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.372968    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f80ac-openshift-default-dockercfg-5qbv4\""
Oct 30 18:54:24.375735 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.375714    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"]
Oct 30 18:54:24.414466 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.414434    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:24.426500 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.426469    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:24.426670 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.426504    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.426670 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.426541    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.426783 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.426759    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.426838 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.426793    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.426889 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.426861    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl4ws\" (UniqueName: \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-kube-api-access-rl4ws\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.427027 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.427006    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.427083 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.427037    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.427134 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.427119    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.427192 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.427178    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.528816 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.528699    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.528816 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.528777    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.528816 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.528808    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529121 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.528847    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529179 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.529135    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529233 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.529185    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529233 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.529206    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529338 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.529245    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529338 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.529317    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl4ws\" (UniqueName: \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-kube-api-access-rl4ws\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529438 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.529393    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529438 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.529423    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529647 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.529431    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529647 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.529603    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.529852 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.529830    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.531548 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.531526    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.532034 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.532017    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.587049 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.587016    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.587258 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.587233    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl4ws\" (UniqueName: \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-kube-api-access-rl4ws\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.588711 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.588687    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:24.588897 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:24.588877    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:54:25.010486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.010428    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"]
Oct 30 18:54:25.011755 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:54:25.011730    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85c4e22_99d6_4616_b2b1_3ce5bd0a8261.slice/crio-1e7693a3f0f0fed64c3be3aeb8e0d9e5451a117b00fbf5ef94bad9064986b293 WatchSource:0}: Error finding container 1e7693a3f0f0fed64c3be3aeb8e0d9e5451a117b00fbf5ef94bad9064986b293: Status 404 returned error can't find the container with id 1e7693a3f0f0fed64c3be3aeb8e0d9e5451a117b00fbf5ef94bad9064986b293
Oct 30 18:54:25.014064 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.014034    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:54:25.014132 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.014095    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:54:25.014132 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.014127    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:54:25.249645 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.249601    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"]
Oct 30 18:54:25.253665 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.253638    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.306348 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.305084    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f5705f-openshift-default-dockercfg-hwfjh\""
Oct 30 18:54:25.309856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.309821    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"]
Oct 30 18:54:25.340648 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.340613    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.340829 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.340669    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.340829 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.340742    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.340829 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.340788    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.341009 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.340904    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.341009 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.340966    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqj9n\" (UniqueName: \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-kube-api-access-wqj9n\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.341100 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.341075    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.341148 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.341136    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.341218 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.341201    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.442831 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.442797    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443027 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.442846    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443027 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.442939    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443151 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.443086    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443206 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.443144    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443206 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.443180    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443299 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.443240    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443352 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.443304    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443408 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.443354    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqj9n\" (UniqueName: \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-kube-api-access-wqj9n\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443534 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.443511    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443618 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.443554    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443684 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.443665    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.443908 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.443818    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.444162 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.444140    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.445675 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.445657    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.445852 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.445838    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.497950 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.497910    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqj9n\" (UniqueName: \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-kube-api-access-wqj9n\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.498401 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.498378    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.568658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.567604    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:25.597051 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.597012    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" event={"ID":"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261","Type":"ContainerStarted","Data":"7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae"}
Oct 30 18:54:25.597195 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.597112    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" event={"ID":"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261","Type":"ContainerStarted","Data":"1e7693a3f0f0fed64c3be3aeb8e0d9e5451a117b00fbf5ef94bad9064986b293"}
Oct 30 18:54:25.739844 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:25.739786    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" podStartSLOduration=1.739771433 podStartE2EDuration="1.739771433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:25.738694716 +0000 UTC m=+1060.849427999" watchObservedRunningTime="2026-10-30 18:54:25.739771433 +0000 UTC m=+1060.850504716"
Oct 30 18:54:26.007157 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:26.007125    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"]
Oct 30 18:54:26.007864 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:54:26.007825    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0c0eae_9f97_4972_894d_0ec6e6c2bb6a.slice/crio-9f654144afda55ce0c655464e97994c30e3015aea189ade5547d390f7ba67e1a WatchSource:0}: Error finding container 9f654144afda55ce0c655464e97994c30e3015aea189ade5547d390f7ba67e1a: Status 404 returned error can't find the container with id 9f654144afda55ce0c655464e97994c30e3015aea189ade5547d390f7ba67e1a
Oct 30 18:54:26.010438 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:26.010405    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:54:26.010515 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:26.010470    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:54:26.010515 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:26.010500    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:54:26.593603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:26.592829    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:26.602605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:26.601083    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:54:26.602605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:26.601143    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 18:54:26.611647 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:26.611610    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" event={"ID":"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a","Type":"ContainerStarted","Data":"9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606"}
Oct 30 18:54:26.611811 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:26.611681    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" event={"ID":"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a","Type":"ContainerStarted","Data":"9f654144afda55ce0c655464e97994c30e3015aea189ade5547d390f7ba67e1a"}
Oct 30 18:54:26.775604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:26.775390    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" podStartSLOduration=1.7753700110000001 podStartE2EDuration="1.775370011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:26.775160371 +0000 UTC m=+1061.885893654" watchObservedRunningTime="2026-10-30 18:54:26.775370011 +0000 UTC m=+1061.886103298"
Oct 30 18:54:27.575604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:27.575071    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:27.594602 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:27.581095    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:54:27.594602 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:27.581167    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 18:54:27.594602 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:27.589481    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:54:27.594602 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:27.589525    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 18:54:28.576603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:28.576036    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:54:28.576603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:28.576098    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 18:54:28.592657 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:28.589958    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:54:28.592657 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:28.590022    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 18:54:29.574017 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:29.573970    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:54:29.574601 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:29.574532    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 18:54:29.591927 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:29.591895    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:54:29.592160 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:29.592130    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 18:54:30.574605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:30.573981    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:54:30.574605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:30.574053    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 18:54:30.593610 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:30.592996    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:54:30.593610 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:30.593068    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 18:54:31.572988 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:31.572950    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:54:31.573176 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:31.573012    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 18:54:31.589969 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:31.589936    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 18:54:31.590355 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:31.589991    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 18:54:32.593701 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:32.593669    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:32.594162 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:32.594074    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:32.594784 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:32.594764    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 18:54:32.812016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:32.811964    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:32.813108 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:32.813085    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:32.813843 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:32.813819    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 18:54:41.253337 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:54:41.253294    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:54:53.422567 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:53.422531    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:54:55.230692 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.230653    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"]
Oct 30 18:54:55.236245 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.236220    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.286430 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.286396    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ec8dab-openshift-default-dockercfg-g5szs\""
Oct 30 18:54:55.337098 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.337059    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"]
Oct 30 18:54:55.409881 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.409849    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294c1b7d-6184-49db-9b0d-4c4ab690a647-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.410088 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.409889    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.410088 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.409980    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.410088 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.410053    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.410225 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.410145    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.410225 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.410182    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.410311 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.410237    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.410311 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.410265    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjv2b\" (UniqueName: \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-kube-api-access-tjv2b\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.410389 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.410355    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.511337 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511248    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.511337 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511296    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.511337 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511320    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.511740 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511343    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjv2b\" (UniqueName: \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-kube-api-access-tjv2b\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.511740 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511374    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.511740 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511427    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294c1b7d-6184-49db-9b0d-4c4ab690a647-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.511740 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511449    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.511740 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511488    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.511740 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511523    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.512049 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511896    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.512049 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.511963    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.512049 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.512019    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.512190 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.512086    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.512306 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.512278    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294c1b7d-6184-49db-9b0d-4c4ab690a647-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.513765 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.513740    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.513873 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.513812    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.567760 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.567727    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.567912 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.567784    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjv2b\" (UniqueName: \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-kube-api-access-tjv2b\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:55.850506 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:55.850422    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:56.284694 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:56.284200    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"]
Oct 30 18:54:56.284694 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:54:56.284472    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294c1b7d_6184_49db_9b0d_4c4ab690a647.slice/crio-a3725424fc37001728db27067cafbd449cec30f34aa3660fe9c81c7ed433781c WatchSource:0}: Error finding container a3725424fc37001728db27067cafbd449cec30f34aa3660fe9c81c7ed433781c: Status 404 returned error can't find the container with id a3725424fc37001728db27067cafbd449cec30f34aa3660fe9c81c7ed433781c
Oct 30 18:54:56.286800 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:56.286744    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:54:56.286885 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:56.286812    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:54:56.286885 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:56.286846    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:54:56.797671 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:56.797628    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" event={"ID":"294c1b7d-6184-49db-9b0d-4c4ab690a647","Type":"ContainerStarted","Data":"20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c"}
Oct 30 18:54:56.797887 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:56.797757    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" event={"ID":"294c1b7d-6184-49db-9b0d-4c4ab690a647","Type":"ContainerStarted","Data":"a3725424fc37001728db27067cafbd449cec30f34aa3660fe9c81c7ed433781c"}
Oct 30 18:54:56.852604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:56.850663    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:54:56.918238 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:56.918177    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" podStartSLOduration=1.9181607299999999 podStartE2EDuration="1.91816073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:56.914515985 +0000 UTC m=+1092.025249303" watchObservedRunningTime="2026-10-30 18:54:56.91816073 +0000 UTC m=+1092.028894016"
Oct 30 18:54:57.853604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:57.853031    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:54:57.853604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:57.853095    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 18:54:58.868606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:58.863984    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:54:58.868606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:58.864050    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 18:54:59.867605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:59.850981    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:54:59.867605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:54:59.851042    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 18:55:00.852055 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:00.852011    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:55:00.852673 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:00.852640    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 18:55:00.974072 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:00.974033    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"]
Oct 30 18:55:00.974564 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:00.974533    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy" containerID="cri-o://92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d" gracePeriod=30
Oct 30 18:55:01.870607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:01.866978    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:55:01.870607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:01.867044    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 18:55:03.038755 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:03.038718    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:55:03.039528 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:03.039510    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:55:03.040406 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:03.040379    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 18:55:06.237825 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.237797    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:55:06.275560 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.275481    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-kube-api-access-2kcrf\" (UniqueName: \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-kube-api-access-2kcrf\") pod \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") "
Oct 30 18:55:06.275560 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.275521    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-data\") pod \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") "
Oct 30 18:55:06.275560 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.275559    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-token\" (UniqueName: \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-token\") pod \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") "
Oct 30 18:55:06.275830 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.275658    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-credential-socket\") pod \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") "
Oct 30 18:55:06.275830 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.275704    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-envoy\") pod \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") "
Oct 30 18:55:06.275830 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.275772    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-socket\") pod \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") "
Oct 30 18:55:06.275830 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.275818    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-podinfo\") pod \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") "
Oct 30 18:55:06.276041 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.275843    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-certs\") pod \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") "
Oct 30 18:55:06.276041 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.275874    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istiod-ca-cert\") pod \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\" (UID: \"1d29565c-5b84-42bb-9d1d-99c93c57ab5e\") "
Oct 30 18:55:06.276041 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.275879    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-data" pod "1d29565c-5b84-42bb-9d1d-99c93c57ab5e" (UID: "1d29565c-5b84-42bb-9d1d-99c93c57ab5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:06.276266 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.276107    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-socket" pod "1d29565c-5b84-42bb-9d1d-99c93c57ab5e" (UID: "1d29565c-5b84-42bb-9d1d-99c93c57ab5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:06.276266 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.276122    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-certs" pod "1d29565c-5b84-42bb-9d1d-99c93c57ab5e" (UID: "1d29565c-5b84-42bb-9d1d-99c93c57ab5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:06.276266 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.276216    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-credential-socket" pod "1d29565c-5b84-42bb-9d1d-99c93c57ab5e" (UID: "1d29565c-5b84-42bb-9d1d-99c93c57ab5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:06.276390 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.276368    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.276429 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.276390    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.276429 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.276404    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.276429 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.276419    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.276539 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.276433    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istiod-ca-cert" pod "1d29565c-5b84-42bb-9d1d-99c93c57ab5e" (UID: "1d29565c-5b84-42bb-9d1d-99c93c57ab5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:06.277953 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.277929    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-kube-api-access-2kcrf" pod "1d29565c-5b84-42bb-9d1d-99c93c57ab5e" (UID: "1d29565c-5b84-42bb-9d1d-99c93c57ab5e"). InnerVolumeSpecName "kube-api-access-2kcrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:06.277953 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.277936    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-podinfo" pod "1d29565c-5b84-42bb-9d1d-99c93c57ab5e" (UID: "1d29565c-5b84-42bb-9d1d-99c93c57ab5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:55:06.278190 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.278164    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-token" pod "1d29565c-5b84-42bb-9d1d-99c93c57ab5e" (UID: "1d29565c-5b84-42bb-9d1d-99c93c57ab5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:06.278291 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.278186    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-envoy" pod "1d29565c-5b84-42bb-9d1d-99c93c57ab5e" (UID: "1d29565c-5b84-42bb-9d1d-99c93c57ab5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:06.377817 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.377763    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.377817 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.377805    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.377817 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.377821    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kcrf\" (UniqueName: \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-kube-api-access-2kcrf\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.378141 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.377837    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.378141 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.377851    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d29565c-5b84-42bb-9d1d-99c93c57ab5e-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.861988 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.861954    4391 generic.go:356] "Generic (PLEG): container finished" podID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerID="92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d" exitCode=0
Oct 30 18:55:06.862198 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.862006    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"
Oct 30 18:55:06.862198 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.862043    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" event={"ID":"1d29565c-5b84-42bb-9d1d-99c93c57ab5e","Type":"ContainerDied","Data":"92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d"}
Oct 30 18:55:06.862198 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.862106    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6" event={"ID":"1d29565c-5b84-42bb-9d1d-99c93c57ab5e","Type":"ContainerDied","Data":"c295d6de63afe872b7600c02a902d49ff4c051077c690f03b58e2e093ab86391"}
Oct 30 18:55:06.862198 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.862138    4391 scope.go:122] "RemoveContainer" containerID="92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d"
Oct 30 18:55:06.872836 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.872817    4391 scope.go:122] "RemoveContainer" containerID="cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d"
Oct 30 18:55:06.881748 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.881726    4391 scope.go:122] "RemoveContainer" containerID="92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d"
Oct 30 18:55:06.882038 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:55:06.882009    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d\": container with ID starting with 92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d not found: ID does not exist" containerID="92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d"
Oct 30 18:55:06.882095 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.882048    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d"} err="failed to get container status \"92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d\": rpc error: code = NotFound desc = could not find container \"92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d\": container with ID starting with 92c54da83e55b89ea706bcada970150249a8c62b04028afce5c9daee174b3d1d not found: ID does not exist"
Oct 30 18:55:06.882095 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.882066    4391 scope.go:122] "RemoveContainer" containerID="cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d"
Oct 30 18:55:06.882310 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:55:06.882292    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d\": container with ID starting with cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d not found: ID does not exist" containerID="cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d"
Oct 30 18:55:06.882357 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:06.882316    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d"} err="failed to get container status \"cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d\": rpc error: code = NotFound desc = could not find container \"cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d\": container with ID starting with cc788d90bb41be4865e0def47db818c66bf3c1cfd3798f12619749564394e01d not found: ID does not exist"
Oct 30 18:55:07.019450 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:07.019419    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"]
Oct 30 18:55:07.065829 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:07.065796    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dszf6"]
Oct 30 18:55:09.005739 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:09.005710    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" path="/var/lib/kubelet/pods/1d29565c-5b84-42bb-9d1d-99c93c57ab5e/volumes"
Oct 30 18:55:11.253098 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:55:11.253060    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:55:31.387763 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:31.387674    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:55:39.889286 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.889249    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"]
Oct 30 18:55:39.889751 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.889737    4391 cpu_manager.go:414] "containerMap: removing container" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy"
Oct 30 18:55:39.889800 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.889752    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy"
Oct 30 18:55:39.889800 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.889773    4391 cpu_manager.go:414] "containerMap: removing container" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy"
Oct 30 18:55:39.889800 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.889779    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy"
Oct 30 18:55:39.889906 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.889866    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy"
Oct 30 18:55:39.889906 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.889876    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d29565c-5b84-42bb-9d1d-99c93c57ab5e" containerName="istio-proxy"
Oct 30 18:55:39.893196 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.893177    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:39.971052 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.971019    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:39.971052 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.971056    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:39.971285 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.971091    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:39.971285 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.971230    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24k5j\" (UniqueName: \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-kube-api-access-24k5j\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:39.971285 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.971264    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:39.971394 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.971331    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:39.971394 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.971369    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:39.971394 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.971390    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:39.971522 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.971502    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:39.992698 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:39.992661    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"]
Oct 30 18:55:40.072573 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.072485    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.072573 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.072532    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.072778 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.072647    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.072778 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.072704    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.072778 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.072760    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24k5j\" (UniqueName: \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-kube-api-access-24k5j\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.072915 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.072789    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.072915 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.072846    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.072915 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.072882    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.072915 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.072911    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.073134 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.072962    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.073134 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.073062    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.073249 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.073214    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.073249 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.073239    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.073573 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.073555    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.074965 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.074947    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.075091 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.075075    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.127088 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.127055    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.127289 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.127263    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24k5j\" (UniqueName: \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-kube-api-access-24k5j\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.206198 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.206159    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:40.249943 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.249912    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"]
Oct 30 18:55:40.254511 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.254484    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"]
Oct 30 18:55:40.254678 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.254620    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.258169 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.258146    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.305299 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.305270    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3b6ad6-openshift-default-dockercfg-wmgsn\""
Oct 30 18:55:40.309856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.309826    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"]
Oct 30 18:55:40.311821 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.311794    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-75e91b-openshift-default-dockercfg-mcrrx\""
Oct 30 18:55:40.319490 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.319456    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"]
Oct 30 18:55:40.376679 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.376597    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.376679 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.376663    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7xx\" (UniqueName: \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-kube-api-access-fs7xx\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.376851 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.376733    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.376851 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.376826    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.376915 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.376871    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.376915 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.376902    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6f7fdb-32a8-479b-a026-224b18d9532f-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.376983 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.376971    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.377016 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377002    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.377049 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377028    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.377126 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377045    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.377200 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377128    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.377257 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377199    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.377315 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377285    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.377367 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377319    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.377367 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377356    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.377446 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377403    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.377446 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377431    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82w5\" (UniqueName: \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-kube-api-access-v82w5\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.377532 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.377451    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.478505 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478456    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.478705 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478514    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.478705 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478554    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.478705 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478597    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.478705 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478624    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82w5\" (UniqueName: \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-kube-api-access-v82w5\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.478705 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478657    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.478974 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478739    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.478974 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478799    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7xx\" (UniqueName: \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-kube-api-access-fs7xx\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.478974 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478827    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.478974 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478865    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.478974 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478901    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.478974 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478902    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.478974 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478954    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6f7fdb-32a8-479b-a026-224b18d9532f-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.479211 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.478992    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.479211 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.479044    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.479211 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.479091    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.479211 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.479121    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.479211 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.479156    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.479211 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.479201    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.479407 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.479260    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.479569 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.479549    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.479834 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.479813    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.480113 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.480094    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.480204 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.480183    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.481010 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.480982    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6f7fdb-32a8-479b-a026-224b18d9532f-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.481197 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.479543    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.481597 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.481530    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.481712 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.481691    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.482003 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.481976    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.482091 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.482061    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.482261 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.482242    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.483084 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.483061    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.533901 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.533870    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.534090 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.533950    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7xx\" (UniqueName: \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-kube-api-access-fs7xx\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.534454 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.534412    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82w5\" (UniqueName: \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-kube-api-access-v82w5\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.537885 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.537862    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.579515 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.579479    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:40.583644 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.583613    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"]
Oct 30 18:55:40.584799 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:55:40.584772    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20aeee3_1a1d_43cc_879e_4f0bac0e7601.slice/crio-cf7e77038d9f8472882776922c46e6e5335b1aa7caa91c304c5808089625a6cd WatchSource:0}: Error finding container cf7e77038d9f8472882776922c46e6e5335b1aa7caa91c304c5808089625a6cd: Status 404 returned error can't find the container with id cf7e77038d9f8472882776922c46e6e5335b1aa7caa91c304c5808089625a6cd
Oct 30 18:55:40.590684 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.590651    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:40.590773 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.590715    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:40.590773 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.590743    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:40.593098 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.593072    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:40.980071 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:55:40.980040    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6f7fdb_32a8_479b_a026_224b18d9532f.slice/crio-8c0fb79edcb88a79a93be3856ef7850c9612d28f0f115026bed465ee87196005 WatchSource:0}: Error finding container 8c0fb79edcb88a79a93be3856ef7850c9612d28f0f115026bed465ee87196005: Status 404 returned error can't find the container with id 8c0fb79edcb88a79a93be3856ef7850c9612d28f0f115026bed465ee87196005
Oct 30 18:55:40.980545 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.980519    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"]
Oct 30 18:55:40.982847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.982815    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:40.982959 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.982887    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:40.982959 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:40.982934    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:41.009074 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:55:41.009050    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8651aaf5_b5f7_4bdd_8f00_c1c4e6bda25d.slice/crio-e7f6f9afb94307f3c3720dd10dee6aadca003ca7d674428a2af4bff88d13b9b5 WatchSource:0}: Error finding container e7f6f9afb94307f3c3720dd10dee6aadca003ca7d674428a2af4bff88d13b9b5: Status 404 returned error can't find the container with id e7f6f9afb94307f3c3720dd10dee6aadca003ca7d674428a2af4bff88d13b9b5
Oct 30 18:55:41.009412 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.009386    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"]
Oct 30 18:55:41.011672 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.011645    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:41.011747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.011713    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:41.011747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.011740    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:41.129940 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.129888    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" event={"ID":"e20aeee3-1a1d-43cc-879e-4f0bac0e7601","Type":"ContainerStarted","Data":"7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7"}
Oct 30 18:55:41.130145 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.129970    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" event={"ID":"e20aeee3-1a1d-43cc-879e-4f0bac0e7601","Type":"ContainerStarted","Data":"cf7e77038d9f8472882776922c46e6e5335b1aa7caa91c304c5808089625a6cd"}
Oct 30 18:55:41.134561 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.134528    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" event={"ID":"ea6f7fdb-32a8-479b-a026-224b18d9532f","Type":"ContainerStarted","Data":"173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb"}
Oct 30 18:55:41.134721 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.134596    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" event={"ID":"ea6f7fdb-32a8-479b-a026-224b18d9532f","Type":"ContainerStarted","Data":"8c0fb79edcb88a79a93be3856ef7850c9612d28f0f115026bed465ee87196005"}
Oct 30 18:55:41.138453 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.138421    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" event={"ID":"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d","Type":"ContainerStarted","Data":"44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3"}
Oct 30 18:55:41.138567 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.138470    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" event={"ID":"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d","Type":"ContainerStarted","Data":"e7f6f9afb94307f3c3720dd10dee6aadca003ca7d674428a2af4bff88d13b9b5"}
Oct 30 18:55:41.206276 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.206242    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:41.207692 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.207671    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:55:41.207764 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.207716    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 18:55:41.255555 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:55:41.255471    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:55:41.385297 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.385224    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" podStartSLOduration=2.385203195 podStartE2EDuration="2.385203195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:41.251869313 +0000 UTC m=+1136.362602597" watchObservedRunningTime="2026-10-30 18:55:41.385203195 +0000 UTC m=+1136.495936479"
Oct 30 18:55:41.387471 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.387404    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" podStartSLOduration=1.3873910280000001 podStartE2EDuration="1.387391028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:41.374336576 +0000 UTC m=+1136.485069861" watchObservedRunningTime="2026-10-30 18:55:41.387391028 +0000 UTC m=+1136.498124312"
Oct 30 18:55:41.491854 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.491778    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" podStartSLOduration=1.491762646 podStartE2EDuration="1.491762646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:41.489841888 +0000 UTC m=+1136.600575237" watchObservedRunningTime="2026-10-30 18:55:41.491762646 +0000 UTC m=+1136.602495931"
Oct 30 18:55:41.579814 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.579709    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:41.593334 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:41.593284    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:42.210113 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:42.210067    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:55:42.210760 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:42.210727    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 18:55:42.604668 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:42.592995    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:55:42.604668 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:42.593065    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 18:55:42.604668 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:42.594860    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:55:42.604668 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:42.594910    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:55:43.208606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:43.208169    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:55:43.208606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:43.208231    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 18:55:43.581660 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:43.580927    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:55:43.581660 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:43.580993    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 18:55:43.601607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:43.593915    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:55:43.601607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:43.593976    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:55:44.208606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:44.208040    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:55:44.208606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:44.208120    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 18:55:44.581659 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:44.580894    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:55:44.581659 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:44.580958    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 18:55:44.601729 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:44.601688    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:55:44.602011 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:44.601982    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:55:45.217255 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:45.215028    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:55:45.217255 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:45.215103    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 18:55:45.597644 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:45.597524    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:55:45.598217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:45.598185    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 18:55:45.599010 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:45.598985    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:55:45.599215 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:45.599178    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:55:46.210606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:46.208970    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:55:46.210606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:46.209034    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 18:55:46.585046 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:46.584954    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:55:46.585280 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:46.585250    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 18:55:46.595136 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:46.595100    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:55:46.595400 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:46.595367    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:55:47.206930 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:47.206861    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:55:47.206930 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:47.206925    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 18:55:47.579991 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:47.579898    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:55:47.579991 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:47.579967    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 18:55:47.595075 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:47.595020    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:55:47.595075 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:47.595069    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 18:55:48.265537 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:48.265499    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:48.266498 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:48.266472    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:48.267374 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:48.267350    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 18:55:48.585801 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:48.585724    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:48.586185 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:48.586166    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:48.587013 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:48.586988    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 18:55:48.597633 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:48.597575    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:48.597881 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:48.597848    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:48.598731 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:48.598710    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 18:55:56.273068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:55:56.273009    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:56:11.252873 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:56:11.252835    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:56:34.529144 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:34.529106    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:56:41.254403 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:56:41.254366    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:56:59.396748 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.396655    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"]
Oct 30 18:56:59.402524 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.402500    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.455692 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.455652    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"]
Oct 30 18:56:59.467830 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.467796    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5j9k\" (UniqueName: \"kubernetes.io/projected/7527b17f-8e81-44eb-8d5e-933530939eca-kube-api-access-t5j9k\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.467982 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.467938    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-dshm\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.468076 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.467978    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-model-cache\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.468076 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.468004    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-home\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.468076 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.468060    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-tmp-dir\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.468203 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.468142    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7527b17f-8e81-44eb-8d5e-933530939eca-tls-certs\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.498078 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.498046    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f80ac\"/\"openshift-service-ca.crt\""
Oct 30 18:56:59.498078 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.498057    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f80ac\"/\"kube-root-ca.crt\""
Oct 30 18:56:59.498745 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.498723    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f80ac\"/\"default-dockercfg-dt6dr\""
Oct 30 18:56:59.498873 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.498746    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f80ac\"/\"distinct-w1-5f80ac-kserve-self-signed-certs\""
Oct 30 18:56:59.575122 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.574285    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5j9k\" (UniqueName: \"kubernetes.io/projected/7527b17f-8e81-44eb-8d5e-933530939eca-kube-api-access-t5j9k\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.575122 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.574400    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-dshm\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.575122 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.574434    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-model-cache\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.575122 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.574458    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-home\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.575122 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.574505    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-tmp-dir\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.575122 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.574527    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7527b17f-8e81-44eb-8d5e-933530939eca-tls-certs\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.578096 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.578057    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-model-cache\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.582495 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.582463    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-dshm\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.583009 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.582983    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-home\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.583396 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.583374    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-tmp-dir\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.589655 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.589537    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7527b17f-8e81-44eb-8d5e-933530939eca-tls-certs\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.632773 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.632740    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5j9k\" (UniqueName: \"kubernetes.io/projected/7527b17f-8e81-44eb-8d5e-933530939eca-kube-api-access-t5j9k\") pod \"distinct-w1-5f80ac-kserve-67468c9747-vl6n6\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:56:59.715061 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:56:59.715020    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:57:00.099038 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:00.099012    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"]
Oct 30 18:57:00.100530 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:57:00.100502    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7527b17f_8e81_44eb_8d5e_933530939eca.slice/crio-076564d2ac33406b350d3043e5458123dbbcf5afe7cca6c774ae6892c2dc8da7 WatchSource:0}: Error finding container 076564d2ac33406b350d3043e5458123dbbcf5afe7cca6c774ae6892c2dc8da7: Status 404 returned error can't find the container with id 076564d2ac33406b350d3043e5458123dbbcf5afe7cca6c774ae6892c2dc8da7
Oct 30 18:57:00.878389 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:00.878344    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" event={"ID":"7527b17f-8e81-44eb-8d5e-933530939eca","Type":"ContainerStarted","Data":"01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074"}
Oct 30 18:57:00.878804 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:00.878403    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" event={"ID":"7527b17f-8e81-44eb-8d5e-933530939eca","Type":"ContainerStarted","Data":"076564d2ac33406b350d3043e5458123dbbcf5afe7cca6c774ae6892c2dc8da7"}
Oct 30 18:57:00.986312 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:00.986257    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podStartSLOduration=1.986242922 podStartE2EDuration="1.986242922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:00.985216623 +0000 UTC m=+1216.095949908" watchObservedRunningTime="2026-10-30 18:57:00.986242922 +0000 UTC m=+1216.096976205"
Oct 30 18:57:01.715464 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:01.715428    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:57:03.196835 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:03.196807    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:57:09.715216 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:09.715173    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:57:09.728398 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:09.728368    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:57:09.946051 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:09.946013    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:57:10.252507 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.252470    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"]
Oct 30 18:57:10.256777 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.256755    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.285659 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.285630    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"]
Oct 30 18:57:10.289633 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.289610    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.311385 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.311356    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"]
Oct 30 18:57:10.403323 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.403147    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.403323 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.403215    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.403323 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.403242    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.403323 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.403287    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.403701 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.403346    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sv59\" (UniqueName: \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-kube-api-access-7sv59\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.403701 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.403373    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.403701 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.403615    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.403701 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.403656    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.403701 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.403685    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.404268 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.403717    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.404385 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.404309    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.404385 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.404330    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.404385 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.404348    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.404522 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.404402    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.404594 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.404538    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.404658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.404620    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb90e7fe-ff29-4428-a376-4b5766eac25a-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.404743 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.404705    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4nhf\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-kube-api-access-t4nhf\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.404816 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.404797    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.407616 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.407572    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"]
Oct 30 18:57:10.506225 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506133    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb90e7fe-ff29-4428-a376-4b5766eac25a-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.506225 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506189    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nhf\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-kube-api-access-t4nhf\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.506422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506259    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.506422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506314    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.506422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506355    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.506422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506383    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.506422 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506415    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.506687 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506477    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sv59\" (UniqueName: \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-kube-api-access-7sv59\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.506687 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506506    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.506687 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506550    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.506687 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506600    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.506687 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506631    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.506687 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506662    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.506995 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506702    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.506995 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506728    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.506995 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506753    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.506995 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506783    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.506995 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506829    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.506995 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.506884    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.507316 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.507014    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.507316 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.507040    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb90e7fe-ff29-4428-a376-4b5766eac25a-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.507316 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.507087    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.507474 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.507317    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.507907 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.507543    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.507907 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.507599    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.507907 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.507746    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.507907 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.507864    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.508121 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.507973    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.509206 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.509175    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.509307 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.509265    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.509393 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.509377    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.509622 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.509574    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.560538 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.560497    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4nhf\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-kube-api-access-t4nhf\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.561270 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.561250    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.561793 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.561770    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:10.561910 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.561895    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sv59\" (UniqueName: \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-kube-api-access-7sv59\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.571694 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.571672    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:10.603504 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:10.603470    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:11.010345 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.010315    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"]
Oct 30 18:57:11.011860 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:57:11.011833    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb90e7fe_ff29_4428_a376_4b5766eac25a.slice/crio-3addb6387f6a4eb5320f9aaf1a7705b63ad4c8ea89974f8a8bf62dfed788a4c4 WatchSource:0}: Error finding container 3addb6387f6a4eb5320f9aaf1a7705b63ad4c8ea89974f8a8bf62dfed788a4c4: Status 404 returned error can't find the container with id 3addb6387f6a4eb5320f9aaf1a7705b63ad4c8ea89974f8a8bf62dfed788a4c4
Oct 30 18:57:11.014247 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.014217    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:11.014319 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.014283    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:11.014319 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.014313    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:11.063169 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.063141    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"]
Oct 30 18:57:11.064826 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:57:11.064802    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb5b8ff_d0e5_4e44_943a_35eeee70716f.slice/crio-24980cfff61365a9d1ea9f1cb125ae712bf2c11ce53c45e8c88f2fe70bf50fc5 WatchSource:0}: Error finding container 24980cfff61365a9d1ea9f1cb125ae712bf2c11ce53c45e8c88f2fe70bf50fc5: Status 404 returned error can't find the container with id 24980cfff61365a9d1ea9f1cb125ae712bf2c11ce53c45e8c88f2fe70bf50fc5
Oct 30 18:57:11.067743 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.067704    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:11.067818 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.067793    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:11.067856 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.067840    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:11.256739 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:57:11.256698    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:57:11.965612 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.964635    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" event={"ID":"eb90e7fe-ff29-4428-a376-4b5766eac25a","Type":"ContainerStarted","Data":"2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483"}
Oct 30 18:57:11.965612 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.964706    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" event={"ID":"eb90e7fe-ff29-4428-a376-4b5766eac25a","Type":"ContainerStarted","Data":"3addb6387f6a4eb5320f9aaf1a7705b63ad4c8ea89974f8a8bf62dfed788a4c4"}
Oct 30 18:57:11.988179 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.987278    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" event={"ID":"5fb5b8ff-d0e5-4e44-943a-35eeee70716f","Type":"ContainerStarted","Data":"25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9"}
Oct 30 18:57:11.988179 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:11.987355    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" event={"ID":"5fb5b8ff-d0e5-4e44-943a-35eeee70716f","Type":"ContainerStarted","Data":"24980cfff61365a9d1ea9f1cb125ae712bf2c11ce53c45e8c88f2fe70bf50fc5"}
Oct 30 18:57:12.094337 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:12.094268    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" podStartSLOduration=2.094245567 podStartE2EDuration="2.094245567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:12.07720833 +0000 UTC m=+1227.187941614" watchObservedRunningTime="2026-10-30 18:57:12.094245567 +0000 UTC m=+1227.204978851"
Oct 30 18:57:12.199838 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:12.199769    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" podStartSLOduration=2.199753567 podStartE2EDuration="2.199753567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:12.195542685 +0000 UTC m=+1227.306275971" watchObservedRunningTime="2026-10-30 18:57:12.199753567 +0000 UTC m=+1227.310486850"
Oct 30 18:57:12.598206 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:12.598172    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:12.600618 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:12.600083    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 18:57:12.600618 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:12.600159    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 18:57:12.604309 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:12.604286    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:12.605576 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:12.605551    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:57:12.605682 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:12.605621    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 18:57:13.574603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:13.573995    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 18:57:13.574603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:13.574073    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 18:57:13.605631 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:13.605596    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:57:13.605803 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:13.605645    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 18:57:14.586604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:14.585710    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 18:57:14.586604 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:14.585781    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 18:57:14.604074 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:14.604037    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:57:14.604247 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:14.604096    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 18:57:15.583978 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:15.583942    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 18:57:15.584263 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:15.584235    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 18:57:15.604180 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:15.604141    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:57:15.604558 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:15.604200    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 18:57:16.579789 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:16.577016    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 18:57:16.579789 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:16.577095    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 18:57:16.605570 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:16.605530    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:57:16.606191 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:16.606158    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 18:57:17.572205 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:17.572174    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 18:57:17.572368 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:17.572237    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 18:57:17.604096 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:17.604065    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 18:57:17.604254 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:17.604117    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 18:57:18.589330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:18.589298    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:18.589851 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:18.589714    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:18.590486 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:18.590467    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 18:57:18.620898 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:18.620870    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:18.621987 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:18.621961    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:18.622883 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:18.622862    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 18:57:31.706503 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:31.706461    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:57:40.268096 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.268061    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"]
Oct 30 18:57:40.272127 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.272104    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.349166 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.349040    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b31304-1ee2-4030-88a1-525f32125a43-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.349557 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.349522    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.349784 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.349764    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.349972 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.349955    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.350166 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.350148    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.350389 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.350374    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.350541 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.350526    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgz7g\" (UniqueName: \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-kube-api-access-lgz7g\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.350727 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.350709    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.350940 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.350911    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b31304-1ee2-4030-88a1-525f32125a43-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.377361 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.377324    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"]
Oct 30 18:57:40.452637 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.452599    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.452637 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.452637    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.452868 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.452688    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.452868 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.452710    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgz7g\" (UniqueName: \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-kube-api-access-lgz7g\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.452868 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.452831    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.453041 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.452905    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b31304-1ee2-4030-88a1-525f32125a43-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.453041 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.453000    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b31304-1ee2-4030-88a1-525f32125a43-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.453147 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.453047    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.453147 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.453106    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.453245 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.453159    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.453245 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.453191    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.453379 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.453354    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.453492 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.453467    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.453734 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.453709    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b31304-1ee2-4030-88a1-525f32125a43-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.455004 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.454982    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.455430 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.455409    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b31304-1ee2-4030-88a1-525f32125a43-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.507150 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.507115    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.507293 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.507207    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgz7g\" (UniqueName: \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-kube-api-access-lgz7g\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:40.587026 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:40.586933    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:41.014708 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:41.014680    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"]
Oct 30 18:57:41.015188 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:57:41.015165    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b31304_1ee2_4030_88a1_525f32125a43.slice/crio-fd749b24491f61ba51298a87295aa376bf34e6ded6a7687b9e06d970b86e4d57 WatchSource:0}: Error finding container fd749b24491f61ba51298a87295aa376bf34e6ded6a7687b9e06d970b86e4d57: Status 404 returned error can't find the container with id fd749b24491f61ba51298a87295aa376bf34e6ded6a7687b9e06d970b86e4d57
Oct 30 18:57:41.017480 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:41.017450    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:41.017559 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:41.017516    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:41.017559 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:41.017543    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:41.183778 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:41.183743    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" event={"ID":"17b31304-1ee2-4030-88a1-525f32125a43","Type":"ContainerStarted","Data":"8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a"}
Oct 30 18:57:41.183977 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:41.183795    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" event={"ID":"17b31304-1ee2-4030-88a1-525f32125a43","Type":"ContainerStarted","Data":"fd749b24491f61ba51298a87295aa376bf34e6ded6a7687b9e06d970b86e4d57"}
Oct 30 18:57:41.257366 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:57:41.257328    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:57:41.297950 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:41.297855    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" podStartSLOduration=1.297839508 podStartE2EDuration="1.297839508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:41.297296089 +0000 UTC m=+1256.408029374" watchObservedRunningTime="2026-10-30 18:57:41.297839508 +0000 UTC m=+1256.408572792"
Oct 30 18:57:41.587433 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:41.587341    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:42.588816 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:42.588778    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:42.589284 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:42.588840    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 18:57:43.588217 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:43.588182    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:43.588425 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:43.588242    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 18:57:44.588134 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:44.588094    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:44.588532 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:44.588145    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 18:57:45.587998 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:45.587950    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:45.588180 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:45.588014    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 18:57:46.588261 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:46.588226    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:46.588680 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:46.588284    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 18:57:47.590986 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:47.590947    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:47.591626 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:47.591570    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 18:57:48.616603 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:48.615833    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:48.620602 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:48.617243    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:57:48.620602 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:57:48.618670    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 18:58:01.340028 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:01.339987    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:58:10.254480 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.254399    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"]
Oct 30 18:58:10.258215 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.258196    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.308679 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.308645    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a1617e-openshift-default-dockercfg-fbsx5\""
Oct 30 18:58:10.312900 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.312869    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.313037 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.313012    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.313108 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.313089    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.313152 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.313121    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.313271 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.313235    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.313435 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.313365    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.313571 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.313459    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgqjm\" (UniqueName: \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-kube-api-access-rgqjm\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.313571 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.313530    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.313571 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.313570    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.378862 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.378826    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"]
Oct 30 18:58:10.414536 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.414480    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.414760 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.414554    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.414760 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.414615    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.414760 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.414668    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.414932 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.414760    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.414932 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.414855    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgqjm\" (UniqueName: \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-kube-api-access-rgqjm\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.414932 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.414888    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.414932 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.414929    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.415091 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.414995    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.415158 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.415109    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.415441 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.415413    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.415441 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.415432    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.415780 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.415756    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.416211 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.416187    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.417218 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.417189    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.419575 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.419545    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.481448 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.481413    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.482647 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.482614    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgqjm\" (UniqueName: \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-kube-api-access-rgqjm\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.572076 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.571976    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:10.995083 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.995051    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"]
Oct 30 18:58:10.995514 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:58:10.995489    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c04b3d_6c49_4b39_bf15_88b8c3a66540.slice/crio-c9afa1c8197da66031eb942bde31ebf4b16efc48148d52f3842102f290fb18a7 WatchSource:0}: Error finding container c9afa1c8197da66031eb942bde31ebf4b16efc48148d52f3842102f290fb18a7: Status 404 returned error can't find the container with id c9afa1c8197da66031eb942bde31ebf4b16efc48148d52f3842102f290fb18a7
Oct 30 18:58:10.997767 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.997736    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:10.997839 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.997798    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:10.997839 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:10.997826    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:11.253645 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:58:11.253542    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:58:11.408523 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:11.408490    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" event={"ID":"93c04b3d-6c49-4b39-bf15-88b8c3a66540","Type":"ContainerStarted","Data":"fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969"}
Oct 30 18:58:11.408986 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:11.408540    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" event={"ID":"93c04b3d-6c49-4b39-bf15-88b8c3a66540","Type":"ContainerStarted","Data":"c9afa1c8197da66031eb942bde31ebf4b16efc48148d52f3842102f290fb18a7"}
Oct 30 18:58:11.523860 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:11.523742    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" podStartSLOduration=1.5237270010000001 podStartE2EDuration="1.523727001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:11.520525897 +0000 UTC m=+1286.631259181" watchObservedRunningTime="2026-10-30 18:58:11.523727001 +0000 UTC m=+1286.634460284"
Oct 30 18:58:11.572394 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:11.572364    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:12.574756 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:12.574706    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 18:58:12.575380 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:12.574786    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 18:58:13.572691 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:13.572646    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 18:58:13.572890 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:13.572732    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 18:58:14.573082 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:14.573047    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 18:58:14.573479 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:14.573105    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 18:58:15.572562 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:15.572526    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 18:58:15.572804 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:15.572635    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 18:58:16.576607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:16.575129    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 18:58:16.576607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:16.575206    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 18:58:17.572846 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:17.572798    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 18:58:17.573051 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:17.572885    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 18:58:18.610985 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:18.610946    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:18.611793 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:18.611769    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:18.612648 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:18.612622    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 18:58:25.309725 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.309688    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"]
Oct 30 18:58:25.314927 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.314907    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.339522 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.339491    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.339680 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.339542    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.339728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.339674    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.339728 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.339710    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.339794 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.339743    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.339830 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.339813    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwvr\" (UniqueName: \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-kube-api-access-7kwvr\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.339899 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.339885    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.339977 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.339960    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.340028 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.340016    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.373249 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.373215    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"]
Oct 30 18:58:25.460593 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.460541    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.460886 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.460858    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwvr\" (UniqueName: \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-kube-api-access-7kwvr\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.461011 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.460941    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.461073 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.461045    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.461147 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.461128    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.470857 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.470824    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.471136 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.471113    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.471364 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.471344    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.471628 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.471603    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.472167 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.472143    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.481845 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.473309    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.482100 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.475539    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.482225 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.476130    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.482315 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.469069    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.482408 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.476647    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.491191 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.491157    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.530069 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.530036    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.530360 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.530337    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwvr\" (UniqueName: \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-kube-api-access-7kwvr\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:25.630068 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:25.629977    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:26.108429 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:26.108369    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"]
Oct 30 18:58:26.110138 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:58:26.110102    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483f0f89_fa50_4e5a_a901_60d7cf6bb333.slice/crio-e9e499038f38ed922b7448971fcce95c49d140fc483255a19a2fd11aaf0b8c33 WatchSource:0}: Error finding container e9e499038f38ed922b7448971fcce95c49d140fc483255a19a2fd11aaf0b8c33: Status 404 returned error can't find the container with id e9e499038f38ed922b7448971fcce95c49d140fc483255a19a2fd11aaf0b8c33
Oct 30 18:58:26.112556 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:26.112519    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:26.112667 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:26.112637    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:26.112739 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:26.112685    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:26.528765 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:26.528725    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" event={"ID":"483f0f89-fa50-4e5a-a901-60d7cf6bb333","Type":"ContainerStarted","Data":"a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1"}
Oct 30 18:58:26.529167 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:26.528779    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" event={"ID":"483f0f89-fa50-4e5a-a901-60d7cf6bb333","Type":"ContainerStarted","Data":"e9e499038f38ed922b7448971fcce95c49d140fc483255a19a2fd11aaf0b8c33"}
Oct 30 18:58:26.630193 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:26.630159    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:26.739785 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:26.739725    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" podStartSLOduration=1.7397077410000001 podStartE2EDuration="1.739707741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:26.737112185 +0000 UTC m=+1301.847845467" watchObservedRunningTime="2026-10-30 18:58:26.739707741 +0000 UTC m=+1301.850441023"
Oct 30 18:58:27.632349 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:27.632266    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:58:27.632349 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:27.632325    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:58:28.630527 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:28.630490    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:58:28.630718 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:28.630542    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:58:29.630874 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:29.630839    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:58:29.631267 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:29.630902    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:58:30.631318 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:30.631286    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:58:30.631746 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:30.631360    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:58:31.631197 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:31.631162    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:58:31.631383 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:31.631217    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:58:32.630885 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:32.630848    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 18:58:32.631089 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:32.630925    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 18:58:33.634396 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:33.634366    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:33.634819 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:33.634791    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:33.635400 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:33.635380    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 18:58:34.396713 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:34.396678    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:58:40.294691 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.294653    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"]
Oct 30 18:58:40.298762 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.298744    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.355491 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.355460    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.355697 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.355522    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.355697 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.355631    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.355793 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.355753    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr56b\" (UniqueName: \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-kube-api-access-cr56b\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.355793 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.355788    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.355900 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.355807    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.355955 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.355937    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d90d268-ee2d-49ac-9634-eaa512441be9-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.356049 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.356030    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.356084 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.356060    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.361407 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.361386    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"]
Oct 30 18:58:40.365488 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.365473    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.416723 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.416675    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"]
Oct 30 18:58:40.466126 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.464473    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.466511 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.466493    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.466665 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.466645    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.466761 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.466714    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.466826 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.466759    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.466826 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.466805    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.466931 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.466836    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.466931 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.466885    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.467029 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.466983    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr56b\" (UniqueName: \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-kube-api-access-cr56b\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.467029 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.467009    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.467144 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.467065    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.467144 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.467090    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.467144 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.467116    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zjgh\" (UniqueName: \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-kube-api-access-8zjgh\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.467362 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.467148    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.467362 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.467209    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d90d268-ee2d-49ac-9634-eaa512441be9-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.467465 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.467358    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.467465 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.467388    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.467465 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.467418    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.470847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.468469    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.470847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.468967    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.470847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.469897    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.470847 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.470518    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d90d268-ee2d-49ac-9634-eaa512441be9-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.471478 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.471440    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.471756 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.471741    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.473060 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.473037    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"]
Oct 30 18:58:40.473722 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.473699    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.523488 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.523449    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.523782 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.523757    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr56b\" (UniqueName: \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-kube-api-access-cr56b\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.568761 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.568668    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.568761 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.568725    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.568992 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.568784    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.568992 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.568857    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.568992 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.568904    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zjgh\" (UniqueName: \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-kube-api-access-8zjgh\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.568992 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.568930    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.569187 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.569046    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.569187 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.569068    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.569187 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.569084    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.569187 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.569138    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.569187 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.569153    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.569437 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.569307    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.569437 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.569379    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.569887 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.569864    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.571018 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.570996    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.571380 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.571363    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.613242 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.613217    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:40.623598 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.623562    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.623725 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.623705    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zjgh\" (UniqueName: \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-kube-api-access-8zjgh\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:40.687740 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:40.682026    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:41.048946 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.048917    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"]
Oct 30 18:58:41.049299 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:58:41.049272    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d90d268_ee2d_49ac_9634_eaa512441be9.slice/crio-4607ffcc87d2744f377d4800651ac91cfd00d4b9cb6fc6eb51a3c2014337c9c4 WatchSource:0}: Error finding container 4607ffcc87d2744f377d4800651ac91cfd00d4b9cb6fc6eb51a3c2014337c9c4: Status 404 returned error can't find the container with id 4607ffcc87d2744f377d4800651ac91cfd00d4b9cb6fc6eb51a3c2014337c9c4
Oct 30 18:58:41.051595 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.051541    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:41.051692 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.051651    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:41.051760 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.051699    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:41.097783 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.097757    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"]
Oct 30 18:58:41.099398 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:58:41.099372    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7516c72_37c4_4f4a_aeb6_eb7eb0baee37.slice/crio-b46fdaa02d1a65d6e664b91c1dfb9d2b0515a345547d18e45756d2b0c5f15730 WatchSource:0}: Error finding container b46fdaa02d1a65d6e664b91c1dfb9d2b0515a345547d18e45756d2b0c5f15730: Status 404 returned error can't find the container with id b46fdaa02d1a65d6e664b91c1dfb9d2b0515a345547d18e45756d2b0c5f15730
Oct 30 18:58:41.102118 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.102084    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:41.102293 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.102160    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:41.102293 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.102199    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:41.255518 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:58:41.255478    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:58:41.652634 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.652575    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" event={"ID":"3d90d268-ee2d-49ac-9634-eaa512441be9","Type":"ContainerStarted","Data":"1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c"}
Oct 30 18:58:41.653079 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.652650    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" event={"ID":"3d90d268-ee2d-49ac-9634-eaa512441be9","Type":"ContainerStarted","Data":"4607ffcc87d2744f377d4800651ac91cfd00d4b9cb6fc6eb51a3c2014337c9c4"}
Oct 30 18:58:41.654118 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.654092    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" event={"ID":"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37","Type":"ContainerStarted","Data":"3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba"}
Oct 30 18:58:41.654223 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.654140    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" event={"ID":"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37","Type":"ContainerStarted","Data":"b46fdaa02d1a65d6e664b91c1dfb9d2b0515a345547d18e45756d2b0c5f15730"}
Oct 30 18:58:41.682376 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.682343    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:41.764254 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.764202    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" podStartSLOduration=1.76418947 podStartE2EDuration="1.76418947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:41.763455473 +0000 UTC m=+1316.874188747" watchObservedRunningTime="2026-10-30 18:58:41.76418947 +0000 UTC m=+1316.874922752"
Oct 30 18:58:41.878911 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:41.878853    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" podStartSLOduration=1.878837651 podStartE2EDuration="1.878837651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:41.877906945 +0000 UTC m=+1316.988640231" watchObservedRunningTime="2026-10-30 18:58:41.878837651 +0000 UTC m=+1316.989570934"
Oct 30 18:58:42.614208 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:42.614169    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:42.615477 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:42.615456    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:58:42.615540 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:42.615511    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 18:58:42.687225 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:42.687184    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 18:58:42.687855 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:42.687812    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 18:58:43.613658 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:43.613624    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:58:43.613844 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:43.613687    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 18:58:43.686605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:43.683955    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 18:58:43.686605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:43.684017    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 18:58:44.613661 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:44.613622    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:58:44.614093 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:44.613677    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 18:58:44.696081 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:44.696041    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 18:58:44.696353 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:44.696320    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 18:58:45.613626 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:45.613573    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:58:45.613875 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:45.613835    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 18:58:45.695605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:45.695151    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 18:58:45.695605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:45.695226    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 18:58:46.614055 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:46.614019    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:58:46.614485 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:46.614078    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 18:58:46.685611 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:46.685039    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 18:58:46.685611 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:46.685121    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 18:58:47.614535 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:47.614492    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:58:47.614951 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:47.614559    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 18:58:47.682992 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:47.682950    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 18:58:47.683143 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:47.683014    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 18:58:48.829012 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:48.828975    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:48.830234 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:48.830205    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:48.831019 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:48.831000    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 18:58:48.973714 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:48.973672    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:48.974337 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:48.974313    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:48.975086 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:48.975066    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 18:58:55.456882 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.456848    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"]
Oct 30 18:58:55.462181 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.462156    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.618814 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.618773    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"]
Oct 30 18:58:55.670664 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.653638    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.670664 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.653704    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.670664 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.653735    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbq68\" (UniqueName: \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-kube-api-access-sbq68\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.670664 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.653761    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.670664 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.653844    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.670664 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.653927    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.670664 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.653953    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.670664 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.654044    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.670664 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.654088    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.755792 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.755706    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.755792 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.755758    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.755792 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.755784    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbq68\" (UniqueName: \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-kube-api-access-sbq68\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.756077 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.755808    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.756077 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.755873    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.756077 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.755963    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.756077 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.755989    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.756255 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.756078    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.756255 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.756123    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.756931 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.756513    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.756931 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.756878    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.761685 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.757154    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.762096 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.762039    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.772061 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.772027    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.776183 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.776148    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.892366 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.892289    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbq68\" (UniqueName: \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-kube-api-access-sbq68\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.904996 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.904968    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:55.930922 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:55.930850    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:56.076627 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:56.076492    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:56.527908 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:56.527872    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"]
Oct 30 18:58:56.546091 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:58:56.546061    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec861e9_b8ce_49e6_92b2_ba55d21354c4.slice/crio-c06a354c7d11f541db56ebb5935ec31673e1d5e17abc6516b85ca3539493c68b WatchSource:0}: Error finding container c06a354c7d11f541db56ebb5935ec31673e1d5e17abc6516b85ca3539493c68b: Status 404 returned error can't find the container with id c06a354c7d11f541db56ebb5935ec31673e1d5e17abc6516b85ca3539493c68b
Oct 30 18:58:56.548731 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:56.548697    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:56.548822 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:56.548768    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:56.548822 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:56.548794    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:56.902249 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:56.902159    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" event={"ID":"aec861e9-b8ce-49e6-92b2-ba55d21354c4","Type":"ContainerStarted","Data":"b1c56cffdce3543d5c55e19286e59b4b859940308ade9470b9c6ce9ecd121a87"}
Oct 30 18:58:56.902249 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:56.902218    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" event={"ID":"aec861e9-b8ce-49e6-92b2-ba55d21354c4","Type":"ContainerStarted","Data":"c06a354c7d11f541db56ebb5935ec31673e1d5e17abc6516b85ca3539493c68b"}
Oct 30 18:58:57.028842 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:57.028787    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" podStartSLOduration=2.02877159 podStartE2EDuration="2.02877159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:57.025810743 +0000 UTC m=+1332.136544027" watchObservedRunningTime="2026-10-30 18:58:57.02877159 +0000 UTC m=+1332.139504873"
Oct 30 18:58:57.077135 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:57.077094    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:58:57.079102 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:57.079068    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:58:57.079278 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:57.079148    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 18:58:58.077782 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:58.077737    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:58:58.078289 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:58.077830    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 18:58:59.077198 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:59.077153    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:58:59.077414 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:58:59.077227    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 18:59:00.077509 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:00.077465    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:59:00.078052 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:00.077526    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 18:59:01.080417 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:01.079955    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:59:01.080417 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:01.080030    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 18:59:02.077911 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:02.077872    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:59:02.078121 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:02.077936    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 18:59:03.077746 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:03.077708    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 18:59:03.078203 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:03.077770    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 18:59:04.299200 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:04.299166    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:59:04.300280 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:04.300244    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:59:04.301213 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:04.301178    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 18:59:11.252569 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:59:11.252531    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:59:18.400715 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:18.400679    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 18:59:20.929964 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:20.929912    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:20.951720 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:20.951675    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.111150 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:21.111105    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:59:21.371795 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:21.371677    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.544784 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:21.544737    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.587741 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:21.587693    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.027174 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.027131    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 18:59:22.052287 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.051561    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 18:59:22.076057 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.056268    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:59:22.093763 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.093733    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:59:22.119704 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.115848    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.119704 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.116362    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:59:22.379967 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.379835    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.381270 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.381246    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:59:22.569717 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.569653    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.570551 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.570528    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:59:22.591731 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.591689    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.592333 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.592309    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:59:22.881186 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.863408    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:59:22.911223 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:22.900259    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:59:23.198605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:23.178687    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:23.388606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:23.382176    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 18:59:23.389165 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:23.389137    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:59:23.389377 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:23.389362    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:59:23.571543 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:23.571444    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:24.089621 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:24.074316    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:59:24.114749 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:24.090391    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 30 18:59:24.115427 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:24.115396    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:59:41.256794 ip-10-0-128-92 kubenswrapper[4391]: E1030 18:59:41.256752    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 18:59:50.382607 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:50.378695    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.397024 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:50.396966    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.695804 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:50.693674    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.726813 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:50.726686    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Oct 30 18:59:50.947710 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:50.946418    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:51.218030 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:51.217908    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:51.394612 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:51.379074    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:51.419455 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:51.419410    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 18:59:51.710727 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:51.710687    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:59:51.711084 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:51.711062    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:59:51.742704 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:51.742657    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:51.766412 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:51.766377    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:59:51.963663 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:51.930898    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 18:59:51.981752 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:51.981695    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:51.983214 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:51.983183    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:59:52.219649 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:52.219535    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:52.221785 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:52.221760    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:59:52.405606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:52.395620    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:52.454606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:52.435024    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 18:59:52.809605 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:52.782267    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:52.992878 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:52.992833    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:53.043606 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:52.995179    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:53.044654 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:52.997234    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 18:59:53.178535 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:53.162828    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 30 18:59:53.192492 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:53.192456    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:59:53.399747 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:53.399707    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.467712 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:53.455672    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:53.639914 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:53.639878    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 18:59:53.760349 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:53.760255    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 18:59:53.921637 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:53.921605    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 18:59:54.111705 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:54.107361    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 18:59:54.115600 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:54.113171    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 18:59:54.596155 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:54.596124    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 18:59:55.473761 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.473706    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"]
Oct 30 18:59:55.477994 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.477970    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:55.539692 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.539653    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"]
Oct 30 18:59:55.541377 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.541339    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad8aed8-nvsqh\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:55.541514 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.541497    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad8aed8-nvsqh\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:55.541633 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.541615    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdgl2\" (UniqueName: \"kubernetes.io/projected/b14a6ae5-8be4-4072-bb29-be96e93323b4-kube-api-access-mdgl2\") pod \"maas-api-revoke-sub-5ad8aed8-nvsqh\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:55.573817 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.573783    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:55.573817 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.573809    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:59:55.650574 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.650027    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad8aed8-nvsqh\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:55.650574 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.650147    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdgl2\" (UniqueName: \"kubernetes.io/projected/b14a6ae5-8be4-4072-bb29-be96e93323b4-kube-api-access-mdgl2\") pod \"maas-api-revoke-sub-5ad8aed8-nvsqh\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:55.650574 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.650285    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad8aed8-nvsqh\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:55.652497 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.651163    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad8aed8-nvsqh\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:55.654330 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.654290    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad8aed8-nvsqh\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:55.706311 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.706277    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdgl2\" (UniqueName: \"kubernetes.io/projected/b14a6ae5-8be4-4072-bb29-be96e93323b4-kube-api-access-mdgl2\") pod \"maas-api-revoke-sub-5ad8aed8-nvsqh\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:55.793666 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:55.793550    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 18:59:56.168520 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:56.168485    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"]
Oct 30 18:59:56.168849 ip-10-0-128-92 kubenswrapper[4391]: W1030 18:59:56.168828    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14a6ae5_8be4_4072_bb29_be96e93323b4.slice/crio-9be3b254cbfa1f912e975d09822a196141387caf2e2e09336eb8a14d2a9a3e2d WatchSource:0}: Error finding container 9be3b254cbfa1f912e975d09822a196141387caf2e2e09336eb8a14d2a9a3e2d: Status 404 returned error can't find the container with id 9be3b254cbfa1f912e975d09822a196141387caf2e2e09336eb8a14d2a9a3e2d
Oct 30 18:59:56.170913 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:56.170894    4391 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:59:56.930012 ip-10-0-128-92 kubenswrapper[4391]: I1030 18:59:56.929978    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh" event={"ID":"b14a6ae5-8be4-4072-bb29-be96e93323b4","Type":"ContainerStarted","Data":"9be3b254cbfa1f912e975d09822a196141387caf2e2e09336eb8a14d2a9a3e2d"}
Oct 30 19:00:00.187130 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.187021    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"]
Oct 30 19:00:00.191744 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.191720    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"
Oct 30 19:00:00.200196 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.199936    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"]
Oct 30 19:00:00.204248 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.204230    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"
Oct 30 19:00:00.211925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.211902    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"]
Oct 30 19:00:00.215808 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.215787    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"
Oct 30 19:00:00.220125 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.220098    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248a3519-e84d-4697-b129-419e8cd88282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6\" (UID: \"248a3519-e84d-4697-b129-419e8cd88282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"
Oct 30 19:00:00.220232 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.220137    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhgxz\" (UniqueName: \"kubernetes.io/projected/248a3519-e84d-4697-b129-419e8cd88282-kube-api-access-vhgxz\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6\" (UID: \"248a3519-e84d-4697-b129-419e8cd88282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"
Oct 30 19:00:00.272308 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.272281    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"]
Oct 30 19:00:00.320102 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.320074    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"]
Oct 30 19:00:00.322693 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.322559    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8f87\" (UniqueName: \"kubernetes.io/projected/d65e75af-b9e1-48cd-832e-06cea16061e4-kube-api-access-l8f87\") pod \"maas-api-key-cleanup-29889780-7n6jq\" (UID: \"d65e75af-b9e1-48cd-832e-06cea16061e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"
Oct 30 19:00:00.322866 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.322806    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248a3519-e84d-4697-b129-419e8cd88282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6\" (UID: \"248a3519-e84d-4697-b129-419e8cd88282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"
Oct 30 19:00:00.322866 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.322834    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhgxz\" (UniqueName: \"kubernetes.io/projected/248a3519-e84d-4697-b129-419e8cd88282-kube-api-access-vhgxz\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6\" (UID: \"248a3519-e84d-4697-b129-419e8cd88282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"
Oct 30 19:00:00.322983 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.322876    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65e75af-b9e1-48cd-832e-06cea16061e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-7n6jq\" (UID: \"d65e75af-b9e1-48cd-832e-06cea16061e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"
Oct 30 19:00:00.322983 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.322911    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac75c28-45f5-488a-9222-9d3b938d247e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm\" (UID: \"3ac75c28-45f5-488a-9222-9d3b938d247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"
Oct 30 19:00:00.322983 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.322942    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7khs4\" (UniqueName: \"kubernetes.io/projected/3ac75c28-45f5-488a-9222-9d3b938d247e-kube-api-access-7khs4\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm\" (UID: \"3ac75c28-45f5-488a-9222-9d3b938d247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"
Oct 30 19:00:00.323895 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.323868    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248a3519-e84d-4697-b129-419e8cd88282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6\" (UID: \"248a3519-e84d-4697-b129-419e8cd88282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"
Oct 30 19:00:00.326082 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.326053    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"]
Oct 30 19:00:00.381347 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.381316    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhgxz\" (UniqueName: \"kubernetes.io/projected/248a3519-e84d-4697-b129-419e8cd88282-kube-api-access-vhgxz\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6\" (UID: \"248a3519-e84d-4697-b129-419e8cd88282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"
Oct 30 19:00:00.423955 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.423918    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65e75af-b9e1-48cd-832e-06cea16061e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-7n6jq\" (UID: \"d65e75af-b9e1-48cd-832e-06cea16061e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"
Oct 30 19:00:00.423955 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.423961    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac75c28-45f5-488a-9222-9d3b938d247e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm\" (UID: \"3ac75c28-45f5-488a-9222-9d3b938d247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"
Oct 30 19:00:00.424157 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.424027    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7khs4\" (UniqueName: \"kubernetes.io/projected/3ac75c28-45f5-488a-9222-9d3b938d247e-kube-api-access-7khs4\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm\" (UID: \"3ac75c28-45f5-488a-9222-9d3b938d247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"
Oct 30 19:00:00.424202 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.424193    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8f87\" (UniqueName: \"kubernetes.io/projected/d65e75af-b9e1-48cd-832e-06cea16061e4-kube-api-access-l8f87\") pod \"maas-api-key-cleanup-29889780-7n6jq\" (UID: \"d65e75af-b9e1-48cd-832e-06cea16061e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"
Oct 30 19:00:00.424748 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.424721    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac75c28-45f5-488a-9222-9d3b938d247e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm\" (UID: \"3ac75c28-45f5-488a-9222-9d3b938d247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"
Oct 30 19:00:00.424865 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.424750    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65e75af-b9e1-48cd-832e-06cea16061e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-7n6jq\" (UID: \"d65e75af-b9e1-48cd-832e-06cea16061e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"
Oct 30 19:00:00.478699 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.478665    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7khs4\" (UniqueName: \"kubernetes.io/projected/3ac75c28-45f5-488a-9222-9d3b938d247e-kube-api-access-7khs4\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm\" (UID: \"3ac75c28-45f5-488a-9222-9d3b938d247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"
Oct 30 19:00:00.479013 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.478994    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8f87\" (UniqueName: \"kubernetes.io/projected/d65e75af-b9e1-48cd-832e-06cea16061e4-kube-api-access-l8f87\") pod \"maas-api-key-cleanup-29889780-7n6jq\" (UID: \"d65e75af-b9e1-48cd-832e-06cea16061e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"
Oct 30 19:00:00.507515 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.507488    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"
Oct 30 19:00:00.519066 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.519042    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"
Oct 30 19:00:00.529086 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.529058    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"
Oct 30 19:00:00.959648 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.959605    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh" event={"ID":"b14a6ae5-8be4-4072-bb29-be96e93323b4","Type":"ContainerStarted","Data":"bc719d32beffc5a3ef79a8c47bff4482693c8a4bd94de9315db6c1f1dbeed3fe"}
Oct 30 19:00:00.984146 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:00.984118    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"]
Oct 30 19:00:00.984868 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:00:00.984846    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac75c28_45f5_488a_9222_9d3b938d247e.slice/crio-4f5dd5cc08dcd6452dd687ec82117fa0838470576ac655db007c1cf21ed67ef3 WatchSource:0}: Error finding container 4f5dd5cc08dcd6452dd687ec82117fa0838470576ac655db007c1cf21ed67ef3: Status 404 returned error can't find the container with id 4f5dd5cc08dcd6452dd687ec82117fa0838470576ac655db007c1cf21ed67ef3
Oct 30 19:00:01.050357 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.050183    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"]
Oct 30 19:00:01.052369 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:00:01.052328    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65e75af_b9e1_48cd_832e_06cea16061e4.slice/crio-dd9b32548f6b3730d3a4b8ab69b35118917bcf1ba438cc119fd5f41bad5ece93 WatchSource:0}: Error finding container dd9b32548f6b3730d3a4b8ab69b35118917bcf1ba438cc119fd5f41bad5ece93: Status 404 returned error can't find the container with id dd9b32548f6b3730d3a4b8ab69b35118917bcf1ba438cc119fd5f41bad5ece93
Oct 30 19:00:01.053810 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:00:01.053783    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248a3519_e84d_4697_b129_419e8cd88282.slice/crio-f9431aab4d19832d9c54b4da8d9bcb07be55e8ff18750c50f6e715a4fa0d0508 WatchSource:0}: Error finding container f9431aab4d19832d9c54b4da8d9bcb07be55e8ff18750c50f6e715a4fa0d0508: Status 404 returned error can't find the container with id f9431aab4d19832d9c54b4da8d9bcb07be55e8ff18750c50f6e715a4fa0d0508
Oct 30 19:00:01.054505 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.054482    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"]
Oct 30 19:00:01.084879 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.084837    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh" podStartSLOduration=1.988791783 podStartE2EDuration="6.08481923s" totalImagesPullingTime="4.096027447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:01.082824795 +0000 UTC m=+1396.193558075" watchObservedRunningTime="2026-10-30 19:00:01.08481923 +0000 UTC m=+1396.195552513"
Oct 30 19:00:01.968697 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.968656    4391 generic.go:356] "Generic (PLEG): container finished" podID="248a3519-e84d-4697-b129-419e8cd88282" containerID="96cc10a41461071a5d32d9dd39da88d38894c666687cfb0ece0c18aaf06841ea" exitCode=60
Oct 30 19:00:01.969083 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.968783    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" event={"ID":"248a3519-e84d-4697-b129-419e8cd88282","Type":"ContainerDied","Data":"96cc10a41461071a5d32d9dd39da88d38894c666687cfb0ece0c18aaf06841ea"}
Oct 30 19:00:01.969083 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.968848    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" event={"ID":"248a3519-e84d-4697-b129-419e8cd88282","Type":"ContainerStarted","Data":"f9431aab4d19832d9c54b4da8d9bcb07be55e8ff18750c50f6e715a4fa0d0508"}
Oct 30 19:00:01.969083 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.969020    4391 scope.go:122] "RemoveContainer" containerID="96cc10a41461071a5d32d9dd39da88d38894c666687cfb0ece0c18aaf06841ea"
Oct 30 19:00:01.970468 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.970440    4391 generic.go:356] "Generic (PLEG): container finished" podID="b14a6ae5-8be4-4072-bb29-be96e93323b4" containerID="bc719d32beffc5a3ef79a8c47bff4482693c8a4bd94de9315db6c1f1dbeed3fe" exitCode=0
Oct 30 19:00:01.970563 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.970512    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh" event={"ID":"b14a6ae5-8be4-4072-bb29-be96e93323b4","Type":"ContainerDied","Data":"bc719d32beffc5a3ef79a8c47bff4482693c8a4bd94de9315db6c1f1dbeed3fe"}
Oct 30 19:00:01.972338 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.972310    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq" event={"ID":"d65e75af-b9e1-48cd-832e-06cea16061e4","Type":"ContainerStarted","Data":"14c1a6824b5a5af95df64db5959d18b11463d19a70e04c437c004c5addc7238b"}
Oct 30 19:00:01.972447 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.972360    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq" event={"ID":"d65e75af-b9e1-48cd-832e-06cea16061e4","Type":"ContainerStarted","Data":"dd9b32548f6b3730d3a4b8ab69b35118917bcf1ba438cc119fd5f41bad5ece93"}
Oct 30 19:00:01.973933 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.973906    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm" event={"ID":"3ac75c28-45f5-488a-9222-9d3b938d247e","Type":"ContainerStarted","Data":"e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56"}
Oct 30 19:00:01.973993 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.973958    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm" event={"ID":"3ac75c28-45f5-488a-9222-9d3b938d247e","Type":"ContainerStarted","Data":"4f5dd5cc08dcd6452dd687ec82117fa0838470576ac655db007c1cf21ed67ef3"}
Oct 30 19:00:01.974149 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:01.974136    4391 scope.go:122] "RemoveContainer" containerID="e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56"
Oct 30 19:00:02.207533 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.207464    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq" podStartSLOduration=2.207443419 podStartE2EDuration="2.207443419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.204117793 +0000 UTC m=+1397.314851076" watchObservedRunningTime="2026-10-30 19:00:02.207443419 +0000 UTC m=+1397.318176697"
Oct 30 19:00:02.983346 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.983311    4391 generic.go:356] "Generic (PLEG): container finished" podID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerID="14c1a6824b5a5af95df64db5959d18b11463d19a70e04c437c004c5addc7238b" exitCode=60
Oct 30 19:00:02.983806 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.983392    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq" event={"ID":"d65e75af-b9e1-48cd-832e-06cea16061e4","Type":"ContainerDied","Data":"14c1a6824b5a5af95df64db5959d18b11463d19a70e04c437c004c5addc7238b"}
Oct 30 19:00:02.983806 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.983718    4391 scope.go:122] "RemoveContainer" containerID="14c1a6824b5a5af95df64db5959d18b11463d19a70e04c437c004c5addc7238b"
Oct 30 19:00:02.985463 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.985440    4391 generic.go:356] "Generic (PLEG): container finished" podID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerID="e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56" exitCode=60
Oct 30 19:00:02.985463 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.985459    4391 generic.go:356] "Generic (PLEG): container finished" podID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerID="59a4768c4888da466ff7f3d27d9e27aaba24fd4a44d6ae7e2583ebe7c3dfd9fb" exitCode=60
Oct 30 19:00:02.985690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.985564    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm" event={"ID":"3ac75c28-45f5-488a-9222-9d3b938d247e","Type":"ContainerDied","Data":"e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56"}
Oct 30 19:00:02.985690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.985645    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm" event={"ID":"3ac75c28-45f5-488a-9222-9d3b938d247e","Type":"ContainerDied","Data":"59a4768c4888da466ff7f3d27d9e27aaba24fd4a44d6ae7e2583ebe7c3dfd9fb"}
Oct 30 19:00:02.985690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.985675    4391 scope.go:122] "RemoveContainer" containerID="e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56"
Oct 30 19:00:02.985839 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.985789    4391 scope.go:122] "RemoveContainer" containerID="59a4768c4888da466ff7f3d27d9e27aaba24fd4a44d6ae7e2583ebe7c3dfd9fb"
Oct 30 19:00:02.986190 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:02.986062    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm_odh-ai-gateway-infra(3ac75c28-45f5-488a-9222-9d3b938d247e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e"
Oct 30 19:00:02.987741 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:02.987704    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" event={"ID":"248a3519-e84d-4697-b129-419e8cd88282","Type":"ContainerStarted","Data":"abc35f0c9db1a09b4b7fe421407a28474666b8e05a8677bb20b3da875db7a1b8"}
Oct 30 19:00:03.034728 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.034705    4391 scope.go:122] "RemoveContainer" containerID="e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56"
Oct 30 19:00:03.035144 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:03.035124    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56\": container with ID starting with e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56 not found: ID does not exist" containerID="e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56"
Oct 30 19:00:03.035235 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.035157    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56"} err="failed to get container status \"e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56\": rpc error: code = NotFound desc = could not find container \"e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56\": container with ID starting with e37be4d48a1ec78abe32b4ac2fa21061e34cf30bfa4d1f67aae370a5d3f6de56 not found: ID does not exist"
Oct 30 19:00:03.192557 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.192528    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 19:00:03.281000 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.280965    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-xks-ca\") pod \"b14a6ae5-8be4-4072-bb29-be96e93323b4\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") "
Oct 30 19:00:03.281173 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.281113    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-openshift-ca\") pod \"b14a6ae5-8be4-4072-bb29-be96e93323b4\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") "
Oct 30 19:00:03.281250 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.281214    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14a6ae5-8be4-4072-bb29-be96e93323b4-kube-api-access-mdgl2\" (UniqueName: \"kubernetes.io/projected/b14a6ae5-8be4-4072-bb29-be96e93323b4-kube-api-access-mdgl2\") pod \"b14a6ae5-8be4-4072-bb29-be96e93323b4\" (UID: \"b14a6ae5-8be4-4072-bb29-be96e93323b4\") "
Oct 30 19:00:03.281525 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.281494    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-openshift-ca" pod "b14a6ae5-8be4-4072-bb29-be96e93323b4" (UID: "b14a6ae5-8be4-4072-bb29-be96e93323b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:03.281682 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.281654    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-openshift-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:03.283202 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.283178    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-xks-ca" pod "b14a6ae5-8be4-4072-bb29-be96e93323b4" (UID: "b14a6ae5-8be4-4072-bb29-be96e93323b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:03.283277 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.283258    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14a6ae5-8be4-4072-bb29-be96e93323b4-kube-api-access-mdgl2" pod "b14a6ae5-8be4-4072-bb29-be96e93323b4" (UID: "b14a6ae5-8be4-4072-bb29-be96e93323b4"). InnerVolumeSpecName "kube-api-access-mdgl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:03.317232 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.317174    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" podStartSLOduration=3.317159219 podStartE2EDuration="3.317159219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.314210036 +0000 UTC m=+1398.424943318" watchObservedRunningTime="2026-10-30 19:00:03.317159219 +0000 UTC m=+1398.427892543"
Oct 30 19:00:03.383250 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.383210    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdgl2\" (UniqueName: \"kubernetes.io/projected/b14a6ae5-8be4-4072-bb29-be96e93323b4-kube-api-access-mdgl2\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:03.383250 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.383245    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b14a6ae5-8be4-4072-bb29-be96e93323b4-maas-api-xks-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:03.996493 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.996456    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"
Oct 30 19:00:03.996965 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.996457    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh" event={"ID":"b14a6ae5-8be4-4072-bb29-be96e93323b4","Type":"ContainerDied","Data":"9be3b254cbfa1f912e975d09822a196141387caf2e2e09336eb8a14d2a9a3e2d"}
Oct 30 19:00:03.996965 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.996633    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9be3b254cbfa1f912e975d09822a196141387caf2e2e09336eb8a14d2a9a3e2d"
Oct 30 19:00:03.998813 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.998789    4391 generic.go:356] "Generic (PLEG): container finished" podID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerID="a8ee867f713a21b0f4f09beefb35a46852f2a1c3d22b40bc0c8155c7bbdb7ad5" exitCode=60
Oct 30 19:00:03.999633 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.998861    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq" event={"ID":"d65e75af-b9e1-48cd-832e-06cea16061e4","Type":"ContainerDied","Data":"a8ee867f713a21b0f4f09beefb35a46852f2a1c3d22b40bc0c8155c7bbdb7ad5"}
Oct 30 19:00:03.999633 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.998922    4391 scope.go:122] "RemoveContainer" containerID="14c1a6824b5a5af95df64db5959d18b11463d19a70e04c437c004c5addc7238b"
Oct 30 19:00:03.999633 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:03.999138    4391 scope.go:122] "RemoveContainer" containerID="a8ee867f713a21b0f4f09beefb35a46852f2a1c3d22b40bc0c8155c7bbdb7ad5"
Oct 30 19:00:03.999633 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:03.999329    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-7n6jq_odh-ai-gateway-infra(d65e75af-b9e1-48cd-832e-06cea16061e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4"
Oct 30 19:00:04.002683 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:04.002665    4391 scope.go:122] "RemoveContainer" containerID="59a4768c4888da466ff7f3d27d9e27aaba24fd4a44d6ae7e2583ebe7c3dfd9fb"
Oct 30 19:00:04.002902 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:04.002881    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm_odh-ai-gateway-infra(3ac75c28-45f5-488a-9222-9d3b938d247e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e"
Oct 30 19:00:04.004832 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:04.004782    4391 generic.go:356] "Generic (PLEG): container finished" podID="248a3519-e84d-4697-b129-419e8cd88282" containerID="abc35f0c9db1a09b4b7fe421407a28474666b8e05a8677bb20b3da875db7a1b8" exitCode=60
Oct 30 19:00:04.004940 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:04.004845    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" event={"ID":"248a3519-e84d-4697-b129-419e8cd88282","Type":"ContainerDied","Data":"abc35f0c9db1a09b4b7fe421407a28474666b8e05a8677bb20b3da875db7a1b8"}
Oct 30 19:00:04.005056 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:04.005043    4391 scope.go:122] "RemoveContainer" containerID="abc35f0c9db1a09b4b7fe421407a28474666b8e05a8677bb20b3da875db7a1b8"
Oct 30 19:00:04.006177 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:04.005278    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6_odh-ai-gateway-infra(248a3519-e84d-4697-b129-419e8cd88282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" podUID="248a3519-e84d-4697-b129-419e8cd88282"
Oct 30 19:00:04.015198 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:04.015168    4391 scope.go:122] "RemoveContainer" containerID="96cc10a41461071a5d32d9dd39da88d38894c666687cfb0ece0c18aaf06841ea"
Oct 30 19:00:05.016778 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:05.016742    4391 scope.go:122] "RemoveContainer" containerID="abc35f0c9db1a09b4b7fe421407a28474666b8e05a8677bb20b3da875db7a1b8"
Oct 30 19:00:05.017238 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:05.017109    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6_odh-ai-gateway-infra(248a3519-e84d-4697-b129-419e8cd88282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" podUID="248a3519-e84d-4697-b129-419e8cd88282"
Oct 30 19:00:05.019888 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:05.019865    4391 scope.go:122] "RemoveContainer" containerID="a8ee867f713a21b0f4f09beefb35a46852f2a1c3d22b40bc0c8155c7bbdb7ad5"
Oct 30 19:00:05.020181 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:05.020156    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-7n6jq_odh-ai-gateway-infra(d65e75af-b9e1-48cd-832e-06cea16061e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4"
Oct 30 19:00:06.212140 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.212104    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"]
Oct 30 19:00:06.212945 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.212926    4391 cpu_manager.go:414] "containerMap: removing container" podUID="b14a6ae5-8be4-4072-bb29-be96e93323b4" containerName="revoke-keys"
Oct 30 19:00:06.213001 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.212946    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14a6ae5-8be4-4072-bb29-be96e93323b4" containerName="revoke-keys"
Oct 30 19:00:06.213067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.213058    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14a6ae5-8be4-4072-bb29-be96e93323b4" containerName="revoke-keys"
Oct 30 19:00:06.216416 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.216400    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.269021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.268989    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"]
Oct 30 19:00:06.310888 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.310851    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:06.321693 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.321661    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74f04158-l6m74\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.321886 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.321721    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74f04158-l6m74\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.321886 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.321850    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbd9s\" (UniqueName: \"kubernetes.io/projected/b69a8582-95df-4b49-a879-8b6f240c4478-kube-api-access-qbd9s\") pod \"maas-api-revoke-sub-74f04158-l6m74\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.422928 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.422893    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74f04158-l6m74\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.423116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.422948    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74f04158-l6m74\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.423116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.422987    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbd9s\" (UniqueName: \"kubernetes.io/projected/b69a8582-95df-4b49-a879-8b6f240c4478-kube-api-access-qbd9s\") pod \"maas-api-revoke-sub-74f04158-l6m74\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.423764 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.423737    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74f04158-l6m74\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.425268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.425239    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74f04158-l6m74\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.438097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.438073    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"]
Oct 30 19:00:06.442281 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.442263    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.478147 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.478072    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbd9s\" (UniqueName: \"kubernetes.io/projected/b69a8582-95df-4b49-a879-8b6f240c4478-kube-api-access-qbd9s\") pod \"maas-api-revoke-sub-74f04158-l6m74\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.493243 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.493087    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-649d16\"/\"distinct-w6-649d16-kserve-self-signed-certs\""
Oct 30 19:00:06.496914 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.496882    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"]
Oct 30 19:00:06.524850 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.524811    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tls-certs\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.525023 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.524856    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnd5f\" (UniqueName: \"kubernetes.io/projected/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-kube-api-access-qnd5f\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.525076 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.524989    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-home\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.525193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.525161    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-model-cache\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.525316 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.525202    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-dshm\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.525316 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.525233    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tmp-dir\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.533449 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.533422    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:06.645631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.637706    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-home\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.645631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.638486    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-model-cache\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.645631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.638520    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-dshm\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.645631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.638547    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tmp-dir\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.645631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.638805    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tls-certs\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.645631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.638843    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnd5f\" (UniqueName: \"kubernetes.io/projected/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-kube-api-access-qnd5f\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.645631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.638343    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-home\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.645631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.639691    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-model-cache\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.664469 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.661550    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tmp-dir\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.666198 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.666161    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-dshm\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.706935 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.702238    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tls-certs\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.707424 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.707394    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnd5f\" (UniqueName: \"kubernetes.io/projected/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-kube-api-access-qnd5f\") pod \"distinct-w6-649d16-kserve-57654c7947-v6xpp\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:06.755296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:06.755163    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:07.066780 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:07.062060    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"]
Oct 30 19:00:07.175067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:07.173112    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"]
Oct 30 19:00:07.176505 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:00:07.175377    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58ba6f26_346c_4cfa_8af2_5b46713e3c6e.slice/crio-47e501f68fce224cd13d58fc99c60f6c8b1b07cd1762f41bdd8b12323e4ba984 WatchSource:0}: Error finding container 47e501f68fce224cd13d58fc99c60f6c8b1b07cd1762f41bdd8b12323e4ba984: Status 404 returned error can't find the container with id 47e501f68fce224cd13d58fc99c60f6c8b1b07cd1762f41bdd8b12323e4ba984
Oct 30 19:00:08.083424 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:08.083393    4391 generic.go:356] "Generic (PLEG): container finished" podID="b69a8582-95df-4b49-a879-8b6f240c4478" containerID="46ff756e4cfe2b06d2136c2d84c1e25251dc5028874545699d45c3f0673c375f" exitCode=0
Oct 30 19:00:08.083831 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:08.083443    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74" event={"ID":"b69a8582-95df-4b49-a879-8b6f240c4478","Type":"ContainerDied","Data":"46ff756e4cfe2b06d2136c2d84c1e25251dc5028874545699d45c3f0673c375f"}
Oct 30 19:00:08.083831 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:08.083502    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74" event={"ID":"b69a8582-95df-4b49-a879-8b6f240c4478","Type":"ContainerStarted","Data":"3d8b8e2eba18c849cdb73894029f8e9eb0d430bc574ee029de02696886c414fd"}
Oct 30 19:00:08.085015 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:08.084994    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" event={"ID":"58ba6f26-346c-4cfa-8af2-5b46713e3c6e","Type":"ContainerStarted","Data":"9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b"}
Oct 30 19:00:08.085069 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:08.085027    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" event={"ID":"58ba6f26-346c-4cfa-8af2-5b46713e3c6e","Type":"ContainerStarted","Data":"47e501f68fce224cd13d58fc99c60f6c8b1b07cd1762f41bdd8b12323e4ba984"}
Oct 30 19:00:08.300234 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:08.300130    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" podStartSLOduration=2.300114324 podStartE2EDuration="2.300114324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:08.298663906 +0000 UTC m=+1403.409397188" watchObservedRunningTime="2026-10-30 19:00:08.300114324 +0000 UTC m=+1403.410847637"
Oct 30 19:00:08.755539 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:08.755501    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:09.249712 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.249683    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:09.305539 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.305502    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69a8582-95df-4b49-a879-8b6f240c4478-kube-api-access-qbd9s\" (UniqueName: \"kubernetes.io/projected/b69a8582-95df-4b49-a879-8b6f240c4478-kube-api-access-qbd9s\") pod \"b69a8582-95df-4b49-a879-8b6f240c4478\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") "
Oct 30 19:00:09.305713 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.305701    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-xks-ca\") pod \"b69a8582-95df-4b49-a879-8b6f240c4478\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") "
Oct 30 19:00:09.305755 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.305747    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-openshift-ca\") pod \"b69a8582-95df-4b49-a879-8b6f240c4478\" (UID: \"b69a8582-95df-4b49-a879-8b6f240c4478\") "
Oct 30 19:00:09.306131 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.306106    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-openshift-ca" pod "b69a8582-95df-4b49-a879-8b6f240c4478" (UID: "b69a8582-95df-4b49-a879-8b6f240c4478"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:09.307624 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.307600    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69a8582-95df-4b49-a879-8b6f240c4478-kube-api-access-qbd9s" pod "b69a8582-95df-4b49-a879-8b6f240c4478" (UID: "b69a8582-95df-4b49-a879-8b6f240c4478"). InnerVolumeSpecName "kube-api-access-qbd9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:09.307702 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.307631    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-xks-ca" pod "b69a8582-95df-4b49-a879-8b6f240c4478" (UID: "b69a8582-95df-4b49-a879-8b6f240c4478"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:09.407315 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.407280    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-xks-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:09.407459 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.407324    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69a8582-95df-4b49-a879-8b6f240c4478-maas-api-openshift-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:09.407459 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.407341    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbd9s\" (UniqueName: \"kubernetes.io/projected/b69a8582-95df-4b49-a879-8b6f240c4478-kube-api-access-qbd9s\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:09.802978 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:09.802867    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:00:10.099699 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.099623    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"
Oct 30 19:00:10.099699 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.099637    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74" event={"ID":"b69a8582-95df-4b49-a879-8b6f240c4478","Type":"ContainerDied","Data":"3d8b8e2eba18c849cdb73894029f8e9eb0d430bc574ee029de02696886c414fd"}
Oct 30 19:00:10.099699 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.099683    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d8b8e2eba18c849cdb73894029f8e9eb0d430bc574ee029de02696886c414fd"
Oct 30 19:00:10.303731 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.303687    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"]
Oct 30 19:00:10.304698 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.304672    4391 cpu_manager.go:414] "containerMap: removing container" podUID="b69a8582-95df-4b49-a879-8b6f240c4478" containerName="revoke-keys"
Oct 30 19:00:10.304880 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.304864    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69a8582-95df-4b49-a879-8b6f240c4478" containerName="revoke-keys"
Oct 30 19:00:10.305197 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.305171    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69a8582-95df-4b49-a879-8b6f240c4478" containerName="revoke-keys"
Oct 30 19:00:10.309396 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.309378    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"]
Oct 30 19:00:10.309507 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.309490    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.313573 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.313552    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.379076 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.378991    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"]
Oct 30 19:00:10.384256 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.384230    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.429917 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.428710    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.430282 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430235    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.430390 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430284    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37e1504c-56bb-474d-8878-fe512509d229-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.430390 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430309    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37e1504c-56bb-474d-8878-fe512509d229-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.430390 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430338    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.430390 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430366    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zgcm\" (UniqueName: \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-kube-api-access-2zgcm\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.430560 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430418    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.430560 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430446    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.430560 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430487    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.430560 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430528    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.430782 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430558    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.430782 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430646    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.430782 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430686    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.430782 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430747    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thdvb\" (UniqueName: \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-kube-api-access-thdvb\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.430782 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430771    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.431029 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430859    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.431029 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430924    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71505255-630b-4cb2-bfda-5841a99f993b-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.431283 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.430950    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.431361 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.431309    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71505255-630b-4cb2-bfda-5841a99f993b-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.431361 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.431347    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.431445 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.431378    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.431480 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.431457    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmhcl\" (UniqueName: \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-kube-api-access-zmhcl\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.431534 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.431497    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.431534 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.431525    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.431669 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.431564    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.431669 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.431606    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.431669 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.431655    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.441528 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.435148    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"]
Oct 30 19:00:10.441877 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.441648    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"]
Oct 30 19:00:10.536603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.535462    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.536603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.535514    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37e1504c-56bb-474d-8878-fe512509d229-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.536603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.535547    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37e1504c-56bb-474d-8878-fe512509d229-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.572614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.571436    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37e1504c-56bb-474d-8878-fe512509d229-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.572614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.571845    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583207    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.535597    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583314    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zgcm\" (UniqueName: \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-kube-api-access-2zgcm\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583374    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583406    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583451    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583496    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583530    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583611    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583651    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583685    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thdvb\" (UniqueName: \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-kube-api-access-thdvb\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583711    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583739    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583766    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71505255-630b-4cb2-bfda-5841a99f993b-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583790    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583823    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71505255-630b-4cb2-bfda-5841a99f993b-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583863    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583896    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.583988    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmhcl\" (UniqueName: \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-kube-api-access-zmhcl\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.584036    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.584067    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.584120    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.584152    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.584186    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.588709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.584266    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.592957 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.592927    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.606118    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.647184    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37e1504c-56bb-474d-8878-fe512509d229-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.647642    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.653256    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71505255-630b-4cb2-bfda-5841a99f993b-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.653558    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.654522    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.659098    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.664236    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.664481    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.665270    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.665496    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.666020    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.666246    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.666641    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.667156    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.669679    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.692855    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.693158    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.693846    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zgcm\" (UniqueName: \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-kube-api-access-2zgcm\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.702605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.697111    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.817659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.797217    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:10.895330 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.895285    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71505255-630b-4cb2-bfda-5841a99f993b-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.895570 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.895387    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"]
Oct 30 19:00:10.917603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.910971    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmhcl\" (UniqueName: \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-kube-api-access-zmhcl\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.926723 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.926493    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:10.929078 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.929010    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thdvb\" (UniqueName: \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-kube-api-access-thdvb\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:10.932914 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:10.932887    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:11.343321 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:11.343166    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:00:11.408456 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.408423    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"]
Oct 30 19:00:11.460611 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.451391    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.460611 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.451477    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.460611 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.451514    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.526055 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.526023    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"]
Oct 30 19:00:11.532326 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.532293    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"]
Oct 30 19:00:11.558165 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.557617    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.558165 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.557702    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.558165 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.557746    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.562783 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:00:11.562736    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71505255_630b_4cb2_bfda_5841a99f993b.slice/crio-672ff40f5618d263e4749497ba5ea98511fbc3947ae57d24f9dc0736d209a059 WatchSource:0}: Error finding container 672ff40f5618d263e4749497ba5ea98511fbc3947ae57d24f9dc0736d209a059: Status 404 returned error can't find the container with id 672ff40f5618d263e4749497ba5ea98511fbc3947ae57d24f9dc0736d209a059
Oct 30 19:00:11.623066 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.622964    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.624422 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.624390    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.624624 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:11.624601    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:12.271539 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:12.271451    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" event={"ID":"71505255-630b-4cb2-bfda-5841a99f993b","Type":"ContainerStarted","Data":"672ff40f5618d263e4749497ba5ea98511fbc3947ae57d24f9dc0736d209a059"}
Oct 30 19:00:12.295355 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:12.293493    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" event={"ID":"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f","Type":"ContainerStarted","Data":"20f253a65e2582e7007b54611e4cadb9a61d2fb0cd636c0dc28b77ade7c212cb"}
Oct 30 19:00:12.332696 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:12.317627    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" event={"ID":"37e1504c-56bb-474d-8878-fe512509d229","Type":"ContainerStarted","Data":"225bddca29502136c91813161d456e3f15a16b01507456c15ebb3ba3a40b6a2f"}
Oct 30 19:00:12.543619 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:12.540832    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podStartSLOduration=2.540809968 podStartE2EDuration="2.540809968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:12.475353684 +0000 UTC m=+1407.586086969" watchObservedRunningTime="2026-10-30 19:00:12.540809968 +0000 UTC m=+1407.651543251"
Oct 30 19:00:12.713606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:12.706764    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podStartSLOduration=2.706728812 podStartE2EDuration="2.706728812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:12.597077012 +0000 UTC m=+1407.707810312" watchObservedRunningTime="2026-10-30 19:00:12.706728812 +0000 UTC m=+1407.817462096"
Oct 30 19:00:12.771646 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:12.768667    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.801763 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:12.801676    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:12.979606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:12.963075    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:13.033723 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.033675    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:00:13.035202 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.035166    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:00:13.160986 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.160679    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.375610 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.363270    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" event={"ID":"37e1504c-56bb-474d-8878-fe512509d229","Type":"ContainerStarted","Data":"b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309"}
Oct 30 19:00:13.384604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.378553    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" event={"ID":"71505255-630b-4cb2-bfda-5841a99f993b","Type":"ContainerStarted","Data":"331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38"}
Oct 30 19:00:13.437681 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.437572    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" event={"ID":"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f","Type":"ContainerStarted","Data":"5eb2d90d0c0a223db16b9c62d0c6470ef7df4a0328eee63149cc09985db76d87"}
Oct 30 19:00:13.514604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.511806    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.514604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.512556    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.538606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.525869    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podStartSLOduration=3.52585103 podStartE2EDuration="3.52585103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:13.522926808 +0000 UTC m=+1408.633660092" watchObservedRunningTime="2026-10-30 19:00:13.52585103 +0000 UTC m=+1408.636584314"
Oct 30 19:00:13.579625 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.579565    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.807972 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.780774    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.807972 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.781179    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 19:00:13.821138 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.820997    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:00:13.821138 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.821063    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 19:00:13.929777 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.929736    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:00:13.930030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.930007    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:00:13.936970 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.935887    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:13.937355 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.937234    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:00:13.937355 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:13.937290    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:00:14.062697 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.050866    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 30 19:00:14.079607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.059235    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.114060 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.059301    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:00:14.150604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.114616    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 19:00:14.150928 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.150902    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 19:00:14.452603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.448938    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Oct 30 19:00:14.620257 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.620215    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:00:14.628344 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.628306    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:00:14.630214 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.630190    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 19:00:14.630401 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.630390    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 19:00:14.838666 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.835969    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.839561 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.839531    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:00:14.839769 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.839744    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 19:00:14.947616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.935098    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.947616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:14.935875    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 19:00:15.005046 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.003703    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:00:15.005046 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.003791    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:00:15.061268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.005357    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:00:15.061597 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.061534    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:00:15.061771 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.038367    4391 scope.go:122] "RemoveContainer" containerID="59a4768c4888da466ff7f3d27d9e27aaba24fd4a44d6ae7e2583ebe7c3dfd9fb"
Oct 30 19:00:15.227126 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.201795    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.227126 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.202123    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.556992 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.541024    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.622273 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.622225    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:00:15.642843 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.642739    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.883721 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.866914    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:00:15.914761 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.914722    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:00:15.918705 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.918659    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 19:00:16.082640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.950521    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:00:16.082640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:15.980730    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:00:16.082640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:16.069563    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:00:16.082640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:16.000729    4391 scope.go:122] "RemoveContainer" containerID="a8ee867f713a21b0f4f09beefb35a46852f2a1c3d22b40bc0c8155c7bbdb7ad5"
Oct 30 19:00:16.082640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:16.082571    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:00:16.082640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:16.082641    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:00:16.204888 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:16.204822    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.205685 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:16.205643    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:00:16.820087 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:16.820040    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.872606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:16.848102    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:00:17.104656 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.102334    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:00:17.104656 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:16.979979    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:17.104656 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.104058    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 19:00:17.109781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:16.954842    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:00:17.109980 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.109954    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 19:00:17.278574 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.095812    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:00:17.278905 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.278878    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:00:17.279132 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.207868    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:17.279402 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.139182    4391 scope.go:122] "RemoveContainer" containerID="abc35f0c9db1a09b4b7fe421407a28474666b8e05a8677bb20b3da875db7a1b8"
Oct 30 19:00:17.280119 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.185686    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:00:17.280280 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.280261    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:00:17.323723 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.323679    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:17.500810 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.500742    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:17.643245 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.642012    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 19:00:17.643245 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.642073    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 19:00:17.643763 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.643742    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:17.645421 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.645403    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 19:00:17.680662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.671834    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm" event={"ID":"3ac75c28-45f5-488a-9222-9d3b938d247e","Type":"ContainerStarted","Data":"f54648da78b3cac31428703344130d96370ce91f712baa8b7b01f4ee10b0df75"}
Oct 30 19:00:17.698457 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.681376    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 19:00:17.731083 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.730064    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 19:00:17.864635 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.858117    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:00:17.864635 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.858184    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 19:00:17.974237 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.969298    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:00:17.989688 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.989644    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:00:17.989889 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.982405    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:00:17.990035 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:17.990017    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:00:18.299659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:18.296188    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm" podStartSLOduration=18.29616916 podStartE2EDuration="18.29616916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:18.12839206 +0000 UTC m=+1413.239125345" watchObservedRunningTime="2026-10-30 19:00:18.29616916 +0000 UTC m=+1413.406902445"
Oct 30 19:00:18.390601 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:18.386196    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:18.925666 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:18.890707    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:00:18.947093 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:18.947047    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 19:00:18.947412 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:18.893943    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq" event={"ID":"d65e75af-b9e1-48cd-832e-06cea16061e4","Type":"ContainerDied","Data":"530206f5d2531c36c67946fe7cf6a28a147f8d85f9262d7e301d7cc744698c3d"}
Oct 30 19:00:18.947723 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:18.947676    4391 scope.go:122] "RemoveContainer" containerID="a8ee867f713a21b0f4f09beefb35a46852f2a1c3d22b40bc0c8155c7bbdb7ad5"
Oct 30 19:00:18.948109 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:18.893904    4391 generic.go:356] "Generic (PLEG): container finished" podID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerID="530206f5d2531c36c67946fe7cf6a28a147f8d85f9262d7e301d7cc744698c3d" exitCode=60
Oct 30 19:00:18.949012 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:18.948993    4391 scope.go:122] "RemoveContainer" containerID="530206f5d2531c36c67946fe7cf6a28a147f8d85f9262d7e301d7cc744698c3d"
Oct 30 19:00:18.950421 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:18.950392    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-7n6jq_odh-ai-gateway-infra(d65e75af-b9e1-48cd-832e-06cea16061e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4"
Oct 30 19:00:19.041801 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.041765    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:00:19.061095 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.061048    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:00:19.146758 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.146724    4391 generic.go:356] "Generic (PLEG): container finished" podID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerID="f54648da78b3cac31428703344130d96370ce91f712baa8b7b01f4ee10b0df75" exitCode=60
Oct 30 19:00:19.312080 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.272090    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:00:19.336105 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.332718    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:00:19.540616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.538262    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm" event={"ID":"3ac75c28-45f5-488a-9222-9d3b938d247e","Type":"ContainerDied","Data":"f54648da78b3cac31428703344130d96370ce91f712baa8b7b01f4ee10b0df75"}
Oct 30 19:00:19.540616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.538329    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" event={"ID":"248a3519-e84d-4697-b129-419e8cd88282","Type":"ContainerStarted","Data":"f0f294ce931176f640a9fa05f674e89537af27435405e62fae26671604823322"}
Oct 30 19:00:19.540616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.538363    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"]
Oct 30 19:00:19.540616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.538871    4391 scope.go:122] "RemoveContainer" containerID="59a4768c4888da466ff7f3d27d9e27aaba24fd4a44d6ae7e2583ebe7c3dfd9fb"
Oct 30 19:00:19.824607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.824457    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:00:19.824819 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.824664    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 19:00:19.941141 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.936396    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:00:19.941141 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.936460    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:00:19.944546 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.944517    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:00:19.944797 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:19.944769    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:00:20.432605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.432013    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"]
Oct 30 19:00:20.444221 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.444188    4391 generic.go:356] "Generic (PLEG): container finished" podID="248a3519-e84d-4697-b129-419e8cd88282" containerID="f0f294ce931176f640a9fa05f674e89537af27435405e62fae26671604823322" exitCode=60
Oct 30 19:00:20.444479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.444354    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" event={"ID":"248a3519-e84d-4697-b129-419e8cd88282","Type":"ContainerDied","Data":"f0f294ce931176f640a9fa05f674e89537af27435405e62fae26671604823322"}
Oct 30 19:00:20.444479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.444420    4391 scope.go:122] "RemoveContainer" containerID="abc35f0c9db1a09b4b7fe421407a28474666b8e05a8677bb20b3da875db7a1b8"
Oct 30 19:00:20.446243 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.446219    4391 scope.go:122] "RemoveContainer" containerID="f0f294ce931176f640a9fa05f674e89537af27435405e62fae26671604823322"
Oct 30 19:00:20.465609 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:20.465179    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6_odh-ai-gateway-infra(248a3519-e84d-4697-b129-419e8cd88282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" podUID="248a3519-e84d-4697-b129-419e8cd88282"
Oct 30 19:00:20.535759 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.535730    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"
Oct 30 19:00:20.634272 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.634173    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac75c28-45f5-488a-9222-9d3b938d247e-kube-api-access-7khs4\" (UniqueName: \"kubernetes.io/projected/3ac75c28-45f5-488a-9222-9d3b938d247e-kube-api-access-7khs4\") pod \"3ac75c28-45f5-488a-9222-9d3b938d247e\" (UID: \"3ac75c28-45f5-488a-9222-9d3b938d247e\") "
Oct 30 19:00:20.634567 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.634540    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac75c28-45f5-488a-9222-9d3b938d247e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac75c28-45f5-488a-9222-9d3b938d247e-maas-api-service-ca\") pod \"3ac75c28-45f5-488a-9222-9d3b938d247e\" (UID: \"3ac75c28-45f5-488a-9222-9d3b938d247e\") "
Oct 30 19:00:20.637624 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.636261    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac75c28-45f5-488a-9222-9d3b938d247e-maas-api-service-ca" pod "3ac75c28-45f5-488a-9222-9d3b938d247e" (UID: "3ac75c28-45f5-488a-9222-9d3b938d247e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:20.639636 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.639571    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac75c28-45f5-488a-9222-9d3b938d247e-kube-api-access-7khs4" pod "3ac75c28-45f5-488a-9222-9d3b938d247e" (UID: "3ac75c28-45f5-488a-9222-9d3b938d247e"). InnerVolumeSpecName "kube-api-access-7khs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:20.738837 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.738804    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7khs4\" (UniqueName: \"kubernetes.io/projected/3ac75c28-45f5-488a-9222-9d3b938d247e-kube-api-access-7khs4\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.739074 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:20.739055    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac75c28-45f5-488a-9222-9d3b938d247e-maas-api-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.042905 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.042867    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:00:21.043330 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.042929    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 19:00:21.043330 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.043196    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:00:21.043330 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.043226    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:00:21.043487 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.043329    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:00:21.043487 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.043369    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:00:21.143182 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.143072    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"]
Oct 30 19:00:21.199928 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.199876    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"
Oct 30 19:00:21.318205 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.318103    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65e75af-b9e1-48cd-832e-06cea16061e4-kube-api-access-l8f87\" (UniqueName: \"kubernetes.io/projected/d65e75af-b9e1-48cd-832e-06cea16061e4-kube-api-access-l8f87\") pod \"d65e75af-b9e1-48cd-832e-06cea16061e4\" (UID: \"d65e75af-b9e1-48cd-832e-06cea16061e4\") "
Oct 30 19:00:21.318589 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.318555    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65e75af-b9e1-48cd-832e-06cea16061e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65e75af-b9e1-48cd-832e-06cea16061e4-maas-api-service-ca\") pod \"d65e75af-b9e1-48cd-832e-06cea16061e4\" (UID: \"d65e75af-b9e1-48cd-832e-06cea16061e4\") "
Oct 30 19:00:21.322789 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.322754    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65e75af-b9e1-48cd-832e-06cea16061e4-maas-api-service-ca" pod "d65e75af-b9e1-48cd-832e-06cea16061e4" (UID: "d65e75af-b9e1-48cd-832e-06cea16061e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:21.325053 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.324826    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65e75af-b9e1-48cd-832e-06cea16061e4-kube-api-access-l8f87" pod "d65e75af-b9e1-48cd-832e-06cea16061e4" (UID: "d65e75af-b9e1-48cd-832e-06cea16061e4"). InnerVolumeSpecName "kube-api-access-l8f87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:21.421547 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.421499    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8f87\" (UniqueName: \"kubernetes.io/projected/d65e75af-b9e1-48cd-832e-06cea16061e4-kube-api-access-l8f87\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.421547 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.421548    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65e75af-b9e1-48cd-832e-06cea16061e4-maas-api-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.498211 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.498177    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"
Oct 30 19:00:21.498672 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.498636    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq" event={"ID":"d65e75af-b9e1-48cd-832e-06cea16061e4","Type":"ContainerDied","Data":"dd9b32548f6b3730d3a4b8ab69b35118917bcf1ba438cc119fd5f41bad5ece93"}
Oct 30 19:00:21.499308 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.499285    4391 scope.go:122] "RemoveContainer" containerID="530206f5d2531c36c67946fe7cf6a28a147f8d85f9262d7e301d7cc744698c3d"
Oct 30 19:00:21.509763 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.509504    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"
Oct 30 19:00:21.509763 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.509516    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm" event={"ID":"3ac75c28-45f5-488a-9222-9d3b938d247e","Type":"ContainerDied","Data":"4f5dd5cc08dcd6452dd687ec82117fa0838470576ac655db007c1cf21ed67ef3"}
Oct 30 19:00:21.520186 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.520122    4391 scope.go:122] "RemoveContainer" containerID="f54648da78b3cac31428703344130d96370ce91f712baa8b7b01f4ee10b0df75"
Oct 30 19:00:21.743663 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.743622    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"]
Oct 30 19:00:21.748306 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.748273    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889780-ncnhm"]
Oct 30 19:00:21.794206 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.794167    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"
Oct 30 19:00:21.800258 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.800209    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:00:21.800500 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.800464    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 19:00:21.844385 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.844355    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"]
Oct 30 19:00:21.844859 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.844826    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248a3519-e84d-4697-b129-419e8cd88282-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248a3519-e84d-4697-b129-419e8cd88282-maas-api-service-ca\") pod \"248a3519-e84d-4697-b129-419e8cd88282\" (UID: \"248a3519-e84d-4697-b129-419e8cd88282\") "
Oct 30 19:00:21.845060 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.844924    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248a3519-e84d-4697-b129-419e8cd88282-kube-api-access-vhgxz\" (UniqueName: \"kubernetes.io/projected/248a3519-e84d-4697-b129-419e8cd88282-kube-api-access-vhgxz\") pod \"248a3519-e84d-4697-b129-419e8cd88282\" (UID: \"248a3519-e84d-4697-b129-419e8cd88282\") "
Oct 30 19:00:21.846990 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.846963    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248a3519-e84d-4697-b129-419e8cd88282-maas-api-service-ca" pod "248a3519-e84d-4697-b129-419e8cd88282" (UID: "248a3519-e84d-4697-b129-419e8cd88282"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:21.851017 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.850955    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-7n6jq"]
Oct 30 19:00:21.855414 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.855386    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248a3519-e84d-4697-b129-419e8cd88282-kube-api-access-vhgxz" pod "248a3519-e84d-4697-b129-419e8cd88282" (UID: "248a3519-e84d-4697-b129-419e8cd88282"). InnerVolumeSpecName "kube-api-access-vhgxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:21.928084 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.928038    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:00:21.928259 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.928104    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:00:21.933744 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.933719    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:00:21.933912 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.933756    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:00:21.947214 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.947184    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248a3519-e84d-4697-b129-419e8cd88282-maas-api-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.947214 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:21.947215    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhgxz\" (UniqueName: \"kubernetes.io/projected/248a3519-e84d-4697-b129-419e8cd88282-kube-api-access-vhgxz\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:22.525683 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:22.525633    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6" event={"ID":"248a3519-e84d-4697-b129-419e8cd88282","Type":"ContainerDied","Data":"f9431aab4d19832d9c54b4da8d9bcb07be55e8ff18750c50f6e715a4fa0d0508"}
Oct 30 19:00:22.526180 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:22.525716    4391 scope.go:122] "RemoveContainer" containerID="f0f294ce931176f640a9fa05f674e89537af27435405e62fae26671604823322"
Oct 30 19:00:22.526180 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:22.525658    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"
Oct 30 19:00:22.697099 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:22.697058    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"]
Oct 30 19:00:22.704296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:22.704256    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889780-5skq6"]
Oct 30 19:00:22.934422 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:22.934388    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:22.934969 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:22.934915    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:22.936030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:22.936003    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:00:23.014074 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:23.014036    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248a3519-e84d-4697-b129-419e8cd88282" path="/var/lib/kubelet/pods/248a3519-e84d-4697-b129-419e8cd88282/volumes"
Oct 30 19:00:23.014719 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:23.014693    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e" path="/var/lib/kubelet/pods/3ac75c28-45f5-488a-9222-9d3b938d247e/volumes"
Oct 30 19:00:23.015305 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:23.015276    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4" path="/var/lib/kubelet/pods/d65e75af-b9e1-48cd-832e-06cea16061e4/volumes"
Oct 30 19:00:23.173071 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:23.172145    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:23.174197 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:23.173558    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:23.174506 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:23.174484    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:00:23.381062 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:23.380971    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:23.381395 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:23.381371    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:23.382703 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:23.382683    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:00:25.289553 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.289514    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"]
Oct 30 19:00:25.290201 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290182    4391 cpu_manager.go:414] "containerMap: removing container" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerName="cleanup"
Oct 30 19:00:25.290201 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290201    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290213    4391 cpu_manager.go:414] "containerMap: removing container" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290220    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290245    4391 cpu_manager.go:414] "containerMap: removing container" podUID="248a3519-e84d-4697-b129-419e8cd88282" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290250    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248a3519-e84d-4697-b129-419e8cd88282" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290262    4391 cpu_manager.go:414] "containerMap: removing container" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290268    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290278    4391 cpu_manager.go:414] "containerMap: removing container" podUID="248a3519-e84d-4697-b129-419e8cd88282" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290282    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248a3519-e84d-4697-b129-419e8cd88282" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290298    4391 cpu_manager.go:414] "containerMap: removing container" podUID="248a3519-e84d-4697-b129-419e8cd88282" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290304    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248a3519-e84d-4697-b129-419e8cd88282" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290319    4391 cpu_manager.go:414] "containerMap: removing container" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290325    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerName="cleanup"
Oct 30 19:00:25.290332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290335    4391 cpu_manager.go:414] "containerMap: removing container" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerName="cleanup"
Oct 30 19:00:25.290724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290340    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerName="cleanup"
Oct 30 19:00:25.290724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290437    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerName="cleanup"
Oct 30 19:00:25.290724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290448    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerName="cleanup"
Oct 30 19:00:25.290724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290455    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerName="cleanup"
Oct 30 19:00:25.290724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290466    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="248a3519-e84d-4697-b129-419e8cd88282" containerName="cleanup"
Oct 30 19:00:25.290724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290474    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerName="cleanup"
Oct 30 19:00:25.290724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290480    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="248a3519-e84d-4697-b129-419e8cd88282" containerName="cleanup"
Oct 30 19:00:25.290724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290486    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="248a3519-e84d-4697-b129-419e8cd88282" containerName="cleanup"
Oct 30 19:00:25.290724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.290492    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerName="cleanup"
Oct 30 19:00:25.295605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.295571    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.348990 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.348960    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"]
Oct 30 19:00:25.416743 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.416697    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.416905 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.416761    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.416905 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.416802    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2aeabe1-d147-468a-855f-dbb4573c0d33-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.416905 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.416871    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.417089 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.416973    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.417089 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.417074    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.417195 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.417103    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rdlq\" (UniqueName: \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-kube-api-access-8rdlq\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.417195 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.417143    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.417289 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.417218    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.518708 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.518670    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.518903 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.518749    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.518903 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.518768    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rdlq\" (UniqueName: \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-kube-api-access-8rdlq\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.518903 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.518791    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.518903 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.518836    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.518903 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.518898    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.519195 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.518930    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.519195 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.518988    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2aeabe1-d147-468a-855f-dbb4573c0d33-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.519195 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.519054    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.519195 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.519088    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.519195 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.519132    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.519195 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.519188    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.519513 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.519314    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.519808 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.519782    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2aeabe1-d147-468a-855f-dbb4573c0d33-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.521098 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.521080    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.521518 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.521496    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.573803 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.573731    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rdlq\" (UniqueName: \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-kube-api-access-8rdlq\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.573803 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.573735    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:25.613674 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:25.613639    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:26.042420 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:26.042389    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"]
Oct 30 19:00:26.042647 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:00:26.042621    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2aeabe1_d147_468a_855f_dbb4573c0d33.slice/crio-42f41ded44549ba1cab55232005f7cde7cb44c67f14c5342979654465bab205c WatchSource:0}: Error finding container 42f41ded44549ba1cab55232005f7cde7cb44c67f14c5342979654465bab205c: Status 404 returned error can't find the container with id 42f41ded44549ba1cab55232005f7cde7cb44c67f14c5342979654465bab205c
Oct 30 19:00:26.044981 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:26.044954    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:26.045050 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:26.045017    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:26.045083 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:26.045056    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:26.593436 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:26.593388    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" event={"ID":"e2aeabe1-d147-468a-855f-dbb4573c0d33","Type":"ContainerStarted","Data":"3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53"}
Oct 30 19:00:26.594054 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:26.593501    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" event={"ID":"e2aeabe1-d147-468a-855f-dbb4573c0d33","Type":"ContainerStarted","Data":"42f41ded44549ba1cab55232005f7cde7cb44c67f14c5342979654465bab205c"}
Oct 30 19:00:26.613773 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:26.613734    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:26.707447 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:26.707385    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" podStartSLOduration=1.7073701890000001 podStartE2EDuration="1.707370189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:26.705272921 +0000 UTC m=+1421.816006204" watchObservedRunningTime="2026-10-30 19:00:26.707370189 +0000 UTC m=+1421.818103472"
Oct 30 19:00:27.615447 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:27.615412    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:00:27.615923 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:27.615469    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 19:00:28.614892 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:28.614857    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:00:28.615082 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:28.614920    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 19:00:29.614314 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:29.614278    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:00:29.614725 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:29.614339    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 19:00:30.614018 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:30.613983    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:00:30.614177 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:30.614063    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 19:00:31.614922 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:31.614879    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:00:31.615406 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:31.614937    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 19:00:32.614496 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:32.614456    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:00:32.614725 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:32.614525    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 19:00:33.618505 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:33.618475    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:33.619010 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:33.618980    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:33.619643 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:33.619624    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:00:34.209432 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.209392    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"]
Oct 30 19:00:34.210109 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.210091    4391 cpu_manager.go:414] "containerMap: removing container" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerName="cleanup"
Oct 30 19:00:34.210187 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.210109    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65e75af-b9e1-48cd-832e-06cea16061e4" containerName="cleanup"
Oct 30 19:00:34.210242 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.210231    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac75c28-45f5-488a-9222-9d3b938d247e" containerName="cleanup"
Oct 30 19:00:34.215934 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.215916    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.267270 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.267240    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"]
Oct 30 19:00:34.273140 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.273102    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:00:34.273307 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.273187    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:34.367992 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.367955    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87003c5d-gccvk\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.368177 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.368077    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przvc\" (UniqueName: \"kubernetes.io/projected/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-kube-api-access-przvc\") pod \"maas-api-revoke-sub-87003c5d-gccvk\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.368247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.368170    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87003c5d-gccvk\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.469233 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.469156    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przvc\" (UniqueName: \"kubernetes.io/projected/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-kube-api-access-przvc\") pod \"maas-api-revoke-sub-87003c5d-gccvk\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.469233 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.469210    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87003c5d-gccvk\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.469415 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.469385    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87003c5d-gccvk\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.469840 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.469809    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87003c5d-gccvk\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.471686 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.471664    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87003c5d-gccvk\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.524309 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.524271    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-przvc\" (UniqueName: \"kubernetes.io/projected/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-kube-api-access-przvc\") pod \"maas-api-revoke-sub-87003c5d-gccvk\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.530783 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.530756    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:34.912845 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:34.912816    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"]
Oct 30 19:00:34.913196 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:00:34.912983    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fac5dc_9c53_4bda_a3e6_a1da7809fd9e.slice/crio-db22669723805c41e511d068e49996a5f481d7dcd19c298d7dc92b0276338138 WatchSource:0}: Error finding container db22669723805c41e511d068e49996a5f481d7dcd19c298d7dc92b0276338138: Status 404 returned error can't find the container with id db22669723805c41e511d068e49996a5f481d7dcd19c298d7dc92b0276338138
Oct 30 19:00:35.670865 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:35.670828    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk" event={"ID":"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e","Type":"ContainerStarted","Data":"0861c24ae730fb5f17d04328b7e39f89c1227b92212808d87f545a0732d59376"}
Oct 30 19:00:35.671125 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:35.670893    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk" event={"ID":"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e","Type":"ContainerStarted","Data":"db22669723805c41e511d068e49996a5f481d7dcd19c298d7dc92b0276338138"}
Oct 30 19:00:35.784365 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:35.784307    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk" podStartSLOduration=1.784293994 podStartE2EDuration="1.784293994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:35.780202976 +0000 UTC m=+1430.890936261" watchObservedRunningTime="2026-10-30 19:00:35.784293994 +0000 UTC m=+1430.895027277"
Oct 30 19:00:36.680145 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:36.680110    4391 generic.go:356] "Generic (PLEG): container finished" podID="c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e" containerID="0861c24ae730fb5f17d04328b7e39f89c1227b92212808d87f545a0732d59376" exitCode=0
Oct 30 19:00:36.680531 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:36.680174    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk" event={"ID":"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e","Type":"ContainerDied","Data":"0861c24ae730fb5f17d04328b7e39f89c1227b92212808d87f545a0732d59376"}
Oct 30 19:00:37.839517 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:37.839487    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:37.914813 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:37.914781    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-xks-ca\") pod \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") "
Oct 30 19:00:37.914989 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:37.914831    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-kube-api-access-przvc\" (UniqueName: \"kubernetes.io/projected/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-kube-api-access-przvc\") pod \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") "
Oct 30 19:00:37.914989 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:37.914895    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-openshift-ca\") pod \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\" (UID: \"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e\") "
Oct 30 19:00:37.915307 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:37.915283    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-openshift-ca" pod "c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e" (UID: "c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:37.915509 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:37.915493    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-openshift-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:37.917067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:37.917040    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-kube-api-access-przvc" pod "c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e" (UID: "c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e"). InnerVolumeSpecName "kube-api-access-przvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:37.917067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:37.917052    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-xks-ca" pod "c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e" (UID: "c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:38.017314 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:38.017237    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-maas-api-xks-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:38.017314 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:38.017268    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-przvc\" (UniqueName: \"kubernetes.io/projected/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e-kube-api-access-przvc\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:38.695731 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:38.695697    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk" event={"ID":"c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e","Type":"ContainerDied","Data":"db22669723805c41e511d068e49996a5f481d7dcd19c298d7dc92b0276338138"}
Oct 30 19:00:38.695933 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:38.695756    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db22669723805c41e511d068e49996a5f481d7dcd19c298d7dc92b0276338138"
Oct 30 19:00:38.695933 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:38.695704    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"
Oct 30 19:00:40.897048 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:40.897014    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8867-3ae1ac"]
Oct 30 19:00:40.897677 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:40.897661    4391 cpu_manager.go:414] "containerMap: removing container" podUID="c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e" containerName="revoke-keys"
Oct 30 19:00:40.897722 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:40.897677    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e" containerName="revoke-keys"
Oct 30 19:00:40.897782 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:40.897773    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e" containerName="revoke-keys"
Oct 30 19:00:40.900654 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:40.900634    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-3ae1ac"
Oct 30 19:00:40.956719 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:40.956688    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-3ae1ac"]
Oct 30 19:00:40.997338 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:40.997303    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qk8z9\""
Oct 30 19:00:41.060687 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:41.060649    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws4qm\" (UniqueName: \"kubernetes.io/projected/bf0d7929-08ed-4f9d-a6ec-ad3889630560-kube-api-access-ws4qm\") pod \"test-curl-8867-3ae1ac\" (UID: \"bf0d7929-08ed-4f9d-a6ec-ad3889630560\") " pod="opendatahub/test-curl-8867-3ae1ac"
Oct 30 19:00:41.162649 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:41.162611    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws4qm\" (UniqueName: \"kubernetes.io/projected/bf0d7929-08ed-4f9d-a6ec-ad3889630560-kube-api-access-ws4qm\") pod \"test-curl-8867-3ae1ac\" (UID: \"bf0d7929-08ed-4f9d-a6ec-ad3889630560\") " pod="opendatahub/test-curl-8867-3ae1ac"
Oct 30 19:00:41.222257 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:41.222222    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws4qm\" (UniqueName: \"kubernetes.io/projected/bf0d7929-08ed-4f9d-a6ec-ad3889630560-kube-api-access-ws4qm\") pod \"test-curl-8867-3ae1ac\" (UID: \"bf0d7929-08ed-4f9d-a6ec-ad3889630560\") " pod="opendatahub/test-curl-8867-3ae1ac"
Oct 30 19:00:41.256535 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:41.256498    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:00:41.512500 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:41.512420    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-3ae1ac"
Oct 30 19:00:41.882390 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:41.882360    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-3ae1ac"]
Oct 30 19:00:41.883060 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:00:41.883035    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf0d7929_08ed_4f9d_a6ec_ad3889630560.slice/crio-b334cf0db8eae65e1d65e93dc6f1a5cfe3e1f5f9c200c60b8677ef3d2acd05ee WatchSource:0}: Error finding container b334cf0db8eae65e1d65e93dc6f1a5cfe3e1f5f9c200c60b8677ef3d2acd05ee: Status 404 returned error can't find the container with id b334cf0db8eae65e1d65e93dc6f1a5cfe3e1f5f9c200c60b8677ef3d2acd05ee
Oct 30 19:00:42.722492 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:42.722452    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-3ae1ac" event={"ID":"bf0d7929-08ed-4f9d-a6ec-ad3889630560","Type":"ContainerStarted","Data":"b334cf0db8eae65e1d65e93dc6f1a5cfe3e1f5f9c200c60b8677ef3d2acd05ee"}
Oct 30 19:00:44.738012 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:44.737978    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-3ae1ac" event={"ID":"bf0d7929-08ed-4f9d-a6ec-ad3889630560","Type":"ContainerStarted","Data":"9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e"}
Oct 30 19:00:44.850620 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:44.850553    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8867-3ae1ac" podStartSLOduration=2.577621727 podStartE2EDuration="4.850537275s" totalImagesPullingTime="2.272915548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:44.848323437 +0000 UTC m=+1439.959056733" watchObservedRunningTime="2026-10-30 19:00:44.850537275 +0000 UTC m=+1439.961270557"
Oct 30 19:00:47.028731 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:47.028677    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.202797 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:47.202733    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.203503 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:47.203473    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.332219 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:47.332115    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.482086 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:47.482042    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.635663 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:47.635532    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:00:47.770605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:47.769747    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" podUID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.791230 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:47.788695    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" podUID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.132606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.131706    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:00:48.132606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.132401    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 19:00:48.222606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.221678    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.254616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.248601    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.254616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.254174    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 19:00:48.254616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.254261    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 19:00:48.344605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.339724    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.344605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.341102    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 19:00:48.831613 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.820907    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:00:48.875609 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.833763    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 19:00:48.875609 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.865257    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" podUID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 19:00:48.875609 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.867700    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:48.875609 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.869525    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:00:48.886050 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.886017    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 19:00:48.901897 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.901763    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:48.901897 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:48.901825    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.085146 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.085058    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:00:49.157412 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.157107    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:49.157412 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.157167    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.173334 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.172953    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.215480 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.213740    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 19:00:49.309110 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.284246    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:00:49.382468 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.283682    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.716621 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.695798    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:49.716621 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.695879    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.716621 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.696657    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:49.716621 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.696705    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.716621 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.700666    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:49.716621 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.700730    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.830605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.819460    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:49.830605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.819525    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.890602 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.871699    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.926628 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.921689    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:49.926628 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.921767    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.959648 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.956684    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:49.959648 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:49.956772    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.029865 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.029729    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:50.030146 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.030101    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.261607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.252760    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8867-3ae1ac"]
Oct 30 19:00:50.261607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.253118    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8867-3ae1ac" podUID="bf0d7929-08ed-4f9d-a6ec-ad3889630560" containerName="test-curl-8867-3ae1ac" containerID="cri-o://9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e" gracePeriod=2
Oct 30 19:00:50.321647 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:50.304178    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-3ae1ac\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="bf0d7929-08ed-4f9d-a6ec-ad3889630560" pod="opendatahub/test-curl-8867-3ae1ac"
Oct 30 19:00:50.351751 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.351702    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.352912 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.352881    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:50.353108 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.353083    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.394408 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.394359    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8867-3ae1ac"]
Oct 30 19:00:50.394791 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.394747    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.496607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.471863    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8867-abe217"]
Oct 30 19:00:50.710610 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.697742    4391 cpu_manager.go:414] "containerMap: removing container" podUID="bf0d7929-08ed-4f9d-a6ec-ad3889630560" containerName="test-curl-8867-3ae1ac"
Oct 30 19:00:50.710610 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.697774    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0d7929-08ed-4f9d-a6ec-ad3889630560" containerName="test-curl-8867-3ae1ac"
Oct 30 19:00:50.710610 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.697944    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf0d7929-08ed-4f9d-a6ec-ad3889630560" containerName="test-curl-8867-3ae1ac"
Oct 30 19:00:50.714801 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.714774    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 19:00:50.715367 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.715346    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-abe217"]
Oct 30 19:00:50.716378 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.715054    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-abe217"
Oct 30 19:00:50.737732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.737689    4391 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.811510 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:50.811464    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-3ae1ac\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="bf0d7929-08ed-4f9d-a6ec-ad3889630560" pod="opendatahub/test-curl-8867-3ae1ac"
Oct 30 19:00:50.888803 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.888760    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8x5n\" (UniqueName: \"kubernetes.io/projected/021ebb7b-d7c7-4780-a666-02239e32fbfd-kube-api-access-n8x5n\") pod \"test-curl-8867-abe217\" (UID: \"021ebb7b-d7c7-4780-a666-02239e32fbfd\") " pod="opendatahub/test-curl-8867-abe217"
Oct 30 19:00:50.919606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.892506    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.941543 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.940707    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:00:50.996314 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:50.995794    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8x5n\" (UniqueName: \"kubernetes.io/projected/021ebb7b-d7c7-4780-a666-02239e32fbfd-kube-api-access-n8x5n\") pod \"test-curl-8867-abe217\" (UID: \"021ebb7b-d7c7-4780-a666-02239e32fbfd\") " pod="opendatahub/test-curl-8867-abe217"
Oct 30 19:00:51.148608 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:51.130885    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" podUID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:51.309662 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:51.301984    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-3ae1ac\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="bf0d7929-08ed-4f9d-a6ec-ad3889630560" pod="opendatahub/test-curl-8867-3ae1ac"
Oct 30 19:00:51.309662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:51.304348    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:51.416051 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:51.416006    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:51.439605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:51.425284    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8x5n\" (UniqueName: \"kubernetes.io/projected/021ebb7b-d7c7-4780-a666-02239e32fbfd-kube-api-access-n8x5n\") pod \"test-curl-8867-abe217\" (UID: \"021ebb7b-d7c7-4780-a666-02239e32fbfd\") " pod="opendatahub/test-curl-8867-abe217"
Oct 30 19:00:51.509684 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:51.509654    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 19:00:51.533509 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:51.533473    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-abe217"
Oct 30 19:00:51.569699 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:51.569669    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 19:00:52.313711 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:52.313676    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8867-abe217"]
Oct 30 19:00:52.412520 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:52.412488    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-abe217" event={"ID":"021ebb7b-d7c7-4780-a666-02239e32fbfd","Type":"ContainerStarted","Data":"1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5"}
Oct 30 19:00:52.412724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:52.412528    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8867-abe217" event={"ID":"021ebb7b-d7c7-4780-a666-02239e32fbfd","Type":"ContainerStarted","Data":"338cf4881f121e22dd94620088c0e581daa3fd69821029909125634524d91b06"}
Oct 30 19:00:52.531058 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:52.531012    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8867-abe217" podStartSLOduration=3.5309980420000002 podStartE2EDuration="3.530998042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:52.525929391 +0000 UTC m=+1447.636662666" watchObservedRunningTime="2026-10-30 19:00:52.530998042 +0000 UTC m=+1447.641731325"
Oct 30 19:00:52.724134 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:52.724110    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-3ae1ac"
Oct 30 19:00:52.864935 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:52.864907    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf0d7929-08ed-4f9d-a6ec-ad3889630560-kube-api-access-ws4qm\" (UniqueName: \"kubernetes.io/projected/bf0d7929-08ed-4f9d-a6ec-ad3889630560-kube-api-access-ws4qm\") pod \"bf0d7929-08ed-4f9d-a6ec-ad3889630560\" (UID: \"bf0d7929-08ed-4f9d-a6ec-ad3889630560\") "
Oct 30 19:00:52.867000 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:52.866970    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0d7929-08ed-4f9d-a6ec-ad3889630560-kube-api-access-ws4qm" pod "bf0d7929-08ed-4f9d-a6ec-ad3889630560" (UID: "bf0d7929-08ed-4f9d-a6ec-ad3889630560"). InnerVolumeSpecName "kube-api-access-ws4qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:52.966938 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:52.966882    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws4qm\" (UniqueName: \"kubernetes.io/projected/bf0d7929-08ed-4f9d-a6ec-ad3889630560-kube-api-access-ws4qm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:53.007077 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:53.007044    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf0d7929-08ed-4f9d-a6ec-ad3889630560" path="/var/lib/kubelet/pods/bf0d7929-08ed-4f9d-a6ec-ad3889630560/volumes"
Oct 30 19:00:53.422097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:53.421856    4391 generic.go:356] "Generic (PLEG): container finished" podID="bf0d7929-08ed-4f9d-a6ec-ad3889630560" containerID="9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e" exitCode=137
Oct 30 19:00:53.423163 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:53.423137    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-3ae1ac"
Oct 30 19:00:53.423689 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:53.423668    4391 scope.go:122] "RemoveContainer" containerID="9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e"
Oct 30 19:00:53.435010 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:53.434989    4391 scope.go:122] "RemoveContainer" containerID="9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e"
Oct 30 19:00:53.435283 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:53.435259    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e\": container with ID starting with 9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e not found: ID does not exist" containerID="9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e"
Oct 30 19:00:53.435355 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:53.435296    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e"} err="failed to get container status \"9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e\": rpc error: code = NotFound desc = could not find container \"9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e\": container with ID starting with 9d355679aa0640d7be9f824f2ad146cb0e10b179e599abeff626fac53eddc82e not found: ID does not exist"
Oct 30 19:00:54.085673 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:54.085638    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8867-abe217"]
Oct 30 19:00:54.091243 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:54.091205    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8867-abe217"]
Oct 30 19:00:54.434187 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:54.434129    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8867-abe217" podUID="021ebb7b-d7c7-4780-a666-02239e32fbfd" containerName="test-curl-8867-abe217" containerID="cri-o://1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5" gracePeriod=2
Oct 30 19:00:54.482205 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:54.482163    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-abe217\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="021ebb7b-d7c7-4780-a666-02239e32fbfd" pod="opendatahub/test-curl-8867-abe217"
Oct 30 19:00:55.052718 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:55.052678    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8867-abe217\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="021ebb7b-d7c7-4780-a666-02239e32fbfd" pod="opendatahub/test-curl-8867-abe217"
Oct 30 19:00:56.688281 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:56.688222    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-abe217"
Oct 30 19:00:56.858981 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:56.858946    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021ebb7b-d7c7-4780-a666-02239e32fbfd-kube-api-access-n8x5n\" (UniqueName: \"kubernetes.io/projected/021ebb7b-d7c7-4780-a666-02239e32fbfd-kube-api-access-n8x5n\") pod \"021ebb7b-d7c7-4780-a666-02239e32fbfd\" (UID: \"021ebb7b-d7c7-4780-a666-02239e32fbfd\") "
Oct 30 19:00:56.861460 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:56.861431    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021ebb7b-d7c7-4780-a666-02239e32fbfd-kube-api-access-n8x5n" pod "021ebb7b-d7c7-4780-a666-02239e32fbfd" (UID: "021ebb7b-d7c7-4780-a666-02239e32fbfd"). InnerVolumeSpecName "kube-api-access-n8x5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:56.960944 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:56.960868    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8x5n\" (UniqueName: \"kubernetes.io/projected/021ebb7b-d7c7-4780-a666-02239e32fbfd-kube-api-access-n8x5n\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:57.012420 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:57.007983    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021ebb7b-d7c7-4780-a666-02239e32fbfd" path="/var/lib/kubelet/pods/021ebb7b-d7c7-4780-a666-02239e32fbfd/volumes"
Oct 30 19:00:57.456514 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:57.456477    4391 generic.go:356] "Generic (PLEG): container finished" podID="021ebb7b-d7c7-4780-a666-02239e32fbfd" containerID="1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5" exitCode=137
Oct 30 19:00:57.456808 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:57.456541    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8867-abe217"
Oct 30 19:00:57.456808 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:57.456550    4391 scope.go:122] "RemoveContainer" containerID="1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5"
Oct 30 19:00:57.466133 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:57.466115    4391 scope.go:122] "RemoveContainer" containerID="1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5"
Oct 30 19:00:57.466430 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:00:57.466411    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5\": container with ID starting with 1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5 not found: ID does not exist" containerID="1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5"
Oct 30 19:00:57.466484 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:00:57.466439    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5"} err="failed to get container status \"1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5\": rpc error: code = NotFound desc = could not find container \"1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5\": container with ID starting with 1fbba1d42e18fb69d537e72d04b68cb2ccb8ab46662498de4c689e45c80bb5b5 not found: ID does not exist"
Oct 30 19:01:11.256886 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.256848    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"]
Oct 30 19:01:11.257356 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:01:11.256951    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:01:11.257595 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.257567    4391 cpu_manager.go:414] "containerMap: removing container" podUID="021ebb7b-d7c7-4780-a666-02239e32fbfd" containerName="test-curl-8867-abe217"
Oct 30 19:01:11.257637 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.257597    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021ebb7b-d7c7-4780-a666-02239e32fbfd" containerName="test-curl-8867-abe217"
Oct 30 19:01:11.257730 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.257721    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="021ebb7b-d7c7-4780-a666-02239e32fbfd" containerName="test-curl-8867-abe217"
Oct 30 19:01:11.261029 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.261010    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.343827 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.343780    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.344015 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.343838    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.344015 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.343998    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.344151 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.344047    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwckz\" (UniqueName: \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-kube-api-access-nwckz\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.344216 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.344196    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.344325 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.344307    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.344383 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.344344    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.344442 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.344397    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.344660 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.344639    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.363973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.363942    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"]
Oct 30 19:01:11.394278 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.394245    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-649d16-openshift-default-dockercfg-xgvdr\""
Oct 30 19:01:11.446182 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446148    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.446182 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446192    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.446502 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446385    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.446502 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446434    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwckz\" (UniqueName: \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-kube-api-access-nwckz\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.446663 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446563    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.446729 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446681    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.446799 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446766    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.446853 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446811    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.446853 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446825    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.446950 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446934    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.447017 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.446987    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.447142 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.447119    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.447259 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.447241    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.447369 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.447343    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.449036 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.449015    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.449318 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.449300    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.500861 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.500831    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.501031 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.500938    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwckz\" (UniqueName: \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-kube-api-access-nwckz\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.574440 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.574347    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:11.999422 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:11.999391    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"]
Oct 30 19:01:12.002077 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:12.002036    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:01:12.002195 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:12.002166    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:01:12.002282 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:12.002265    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:01:12.585247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:12.585213    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" event={"ID":"1861d4e6-403e-4ebb-a619-87bc3bb7efb7","Type":"ContainerStarted","Data":"912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214"}
Oct 30 19:01:12.585698 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:12.585265    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" event={"ID":"1861d4e6-403e-4ebb-a619-87bc3bb7efb7","Type":"ContainerStarted","Data":"4a3b8f7c8e51a8cd64f1f8a73c9f97df870ab7a674ee5248617c8fe5aae76cf7"}
Oct 30 19:01:12.705528 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:12.705461    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" podStartSLOduration=1.705443023 podStartE2EDuration="1.705443023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:12.702356376 +0000 UTC m=+1467.813089661" watchObservedRunningTime="2026-10-30 19:01:12.705443023 +0000 UTC m=+1467.816176307"
Oct 30 19:01:13.575012 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:13.574967    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:13.577037 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:13.577006    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:01:13.577173 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:13.577056    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:01:14.575277 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:14.575238    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:01:14.575696 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:14.575301    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:01:15.635741 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:15.635703    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:01:15.636345 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:15.636313    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:01:16.578434 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:16.578037    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:01:16.578434 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:16.578099    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:01:17.574963 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:17.574923    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:01:17.575370 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:17.574983    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:01:18.574982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:18.574940    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:01:18.575396 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:18.575024    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 19:01:19.685566 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:19.685528    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:19.686435 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:19.686408    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:19.687220 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:19.687202    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:01:20.920535 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:20.920492    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:01:24.140604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:24.140454    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" podUID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:24.193616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:24.191872    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:24.242616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:24.235675    4391 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:24.242616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:24.235740    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:32.515664 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.510898    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"]
Oct 30 19:01:32.522607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.521629    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:01:32.557623 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.554936    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"]
Oct 30 19:01:32.577184 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.577146    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-15046cf2\""
Oct 30 19:01:32.577541 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.577520    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-15046cf2-dockercfg-p75n8\""
Oct 30 19:01:32.658626 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.651212    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn\" (UID: \"f6a49b9e-9650-44b1-88c2-1ab6cc59289f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:01:32.670606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.663846    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2nmf\" (UniqueName: \"kubernetes.io/projected/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-kube-api-access-w2nmf\") pod \"payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn\" (UID: \"f6a49b9e-9650-44b1-88c2-1ab6cc59289f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:01:32.781258 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.781157    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2nmf\" (UniqueName: \"kubernetes.io/projected/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-kube-api-access-w2nmf\") pod \"payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn\" (UID: \"f6a49b9e-9650-44b1-88c2-1ab6cc59289f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:01:32.781442 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.781346    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn\" (UID: \"f6a49b9e-9650-44b1-88c2-1ab6cc59289f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:01:32.782362 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.782335    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn\" (UID: \"f6a49b9e-9650-44b1-88c2-1ab6cc59289f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:01:32.928213 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:32.928177    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2nmf\" (UniqueName: \"kubernetes.io/projected/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-kube-api-access-w2nmf\") pod \"payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn\" (UID: \"f6a49b9e-9650-44b1-88c2-1ab6cc59289f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:01:33.165425 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:33.165390    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:01:33.536767 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:01:33.536737    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a49b9e_9650_44b1_88c2_1ab6cc59289f.slice/crio-e7fff72bce925f1d7134bd9a39d10aaa3286a9dd14c6eb720b371d4d04978ecd WatchSource:0}: Error finding container e7fff72bce925f1d7134bd9a39d10aaa3286a9dd14c6eb720b371d4d04978ecd: Status 404 returned error can't find the container with id e7fff72bce925f1d7134bd9a39d10aaa3286a9dd14c6eb720b371d4d04978ecd
Oct 30 19:01:33.537193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:33.536790    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"]
Oct 30 19:01:34.364409 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:34.364373    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn" event={"ID":"f6a49b9e-9650-44b1-88c2-1ab6cc59289f","Type":"ContainerStarted","Data":"21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4"}
Oct 30 19:01:34.364621 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:34.364431    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn" event={"ID":"f6a49b9e-9650-44b1-88c2-1ab6cc59289f","Type":"ContainerStarted","Data":"e7fff72bce925f1d7134bd9a39d10aaa3286a9dd14c6eb720b371d4d04978ecd"}
Oct 30 19:01:34.364621 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:34.364459    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:01:34.476810 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:34.476758    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn" podStartSLOduration=2.476744824 podStartE2EDuration="2.476744824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:34.471652615 +0000 UTC m=+1489.582385901" watchObservedRunningTime="2026-10-30 19:01:34.476744824 +0000 UTC m=+1489.587478106"
Oct 30 19:01:41.253966 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:01:41.253928    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:01:45.382031 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:45.381998    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:01:46.819968 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:46.819937    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"]
Oct 30 19:01:46.824289 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:46.824268    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:46.875224 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:46.875193    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-948bc488-serving-cert\""
Oct 30 19:01:46.926561 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:46.926518    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"]
Oct 30 19:01:46.945320 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:46.945283    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-metrics-certs\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:46.945474 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:46.945400    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhdr\" (UniqueName: \"kubernetes.io/projected/b8c33062-7b1f-46ef-941e-43475212a5ee-kube-api-access-phhdr\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:46.945533 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:46.945505    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-maas-api-tls\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:46.945680 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:46.945660    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c33062-7b1f-46ef-941e-43475212a5ee-openshift-service-ca\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:47.046523 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:47.046480    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c33062-7b1f-46ef-941e-43475212a5ee-openshift-service-ca\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:47.046740 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:47.046678    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-metrics-certs\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:47.046811 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:47.046752    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhdr\" (UniqueName: \"kubernetes.io/projected/b8c33062-7b1f-46ef-941e-43475212a5ee-kube-api-access-phhdr\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:47.046992 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:47.046976    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-maas-api-tls\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:47.047193 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:01:47.047070    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-serving-cert: secret "maas-api-e2e-migrate-948bc488-serving-cert" not found
Oct 30 19:01:47.047320 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:47.047194    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c33062-7b1f-46ef-941e-43475212a5ee-openshift-service-ca\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:47.047320 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:01:47.047250    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-maas-api-tls podName:b8c33062-7b1f-46ef-941e-43475212a5ee nodeName:}" failed. No retries permitted until 2026-10-30 19:01:47.547233369 +0000 UTC m=+1502.657966630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-maas-api-tls") pod "maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww" (UID: "b8c33062-7b1f-46ef-941e-43475212a5ee") : secret "maas-api-e2e-migrate-948bc488-serving-cert" not found
Oct 30 19:01:47.049305 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:47.049281    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-metrics-certs\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:47.101746 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:47.101671    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhdr\" (UniqueName: \"kubernetes.io/projected/b8c33062-7b1f-46ef-941e-43475212a5ee-kube-api-access-phhdr\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:47.562111 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:47.562079    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-maas-api-tls\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:47.564428 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:47.564401    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-maas-api-tls\") pod \"maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:47.739049 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:47.739011    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:48.114133 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:01:48.114100    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c33062_7b1f_46ef_941e_43475212a5ee.slice/crio-9f199b3eedc9d7a6edf6a15f578ba8b3a3515ea72720b31a78c763479a6fc3ee WatchSource:0}: Error finding container 9f199b3eedc9d7a6edf6a15f578ba8b3a3515ea72720b31a78c763479a6fc3ee: Status 404 returned error can't find the container with id 9f199b3eedc9d7a6edf6a15f578ba8b3a3515ea72720b31a78c763479a6fc3ee
Oct 30 19:01:48.114519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:48.114499    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"]
Oct 30 19:01:48.480669 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:48.480626    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww" event={"ID":"b8c33062-7b1f-46ef-941e-43475212a5ee","Type":"ContainerStarted","Data":"9f199b3eedc9d7a6edf6a15f578ba8b3a3515ea72720b31a78c763479a6fc3ee"}
Oct 30 19:01:49.489323 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:49.489274    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww" event={"ID":"b8c33062-7b1f-46ef-941e-43475212a5ee","Type":"ContainerStarted","Data":"648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f"}
Oct 30 19:01:49.490042 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:49.489381    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:49.601220 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:49.601155    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww" podStartSLOduration=2.611949843 podStartE2EDuration="3.601135455s" totalImagesPullingTime="989.185612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:49.597758841 +0000 UTC m=+1504.708492129" watchObservedRunningTime="2026-10-30 19:01:49.601135455 +0000 UTC m=+1504.711868740"
Oct 30 19:01:55.501963 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:55.501928    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:01:56.308880 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.308844    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"]
Oct 30 19:01:56.312398 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.312380    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.366742 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.366407    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"]
Oct 30 19:01:56.367536 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.367505    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wqbk\" (UniqueName: \"kubernetes.io/projected/e678e320-130a-4cb1-a943-51c3467f266e-kube-api-access-4wqbk\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.367745 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.367570    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e678e320-130a-4cb1-a943-51c3467f266e-openshift-service-ca\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.367745 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.367631    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-metrics-certs\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.367745 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.367726    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-maas-api-tls\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.407612 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.407563    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-85b6ee7b-serving-cert\""
Oct 30 19:01:56.469039 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.468998    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wqbk\" (UniqueName: \"kubernetes.io/projected/e678e320-130a-4cb1-a943-51c3467f266e-kube-api-access-4wqbk\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.469238 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.469075    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e678e320-130a-4cb1-a943-51c3467f266e-openshift-service-ca\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.469238 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.469115    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-metrics-certs\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.469238 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.469221    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-maas-api-tls\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.471054 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.471027    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e678e320-130a-4cb1-a943-51c3467f266e-openshift-service-ca\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.471355 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:01:56.471333    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-serving-cert: secret "maas-api-e2e-ait-85b6ee7b-serving-cert" not found
Oct 30 19:01:56.471512 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:01:56.471408    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-maas-api-tls podName:e678e320-130a-4cb1-a943-51c3467f266e nodeName:}" failed. No retries permitted until 2026-10-30 19:01:56.971388519 +0000 UTC m=+1512.082121794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-maas-api-tls") pod "maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz" (UID: "e678e320-130a-4cb1-a943-51c3467f266e") : secret "maas-api-e2e-ait-85b6ee7b-serving-cert" not found
Oct 30 19:01:56.473945 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.473895    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-metrics-certs\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:56.525448 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:56.525417    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wqbk\" (UniqueName: \"kubernetes.io/projected/e678e320-130a-4cb1-a943-51c3467f266e-kube-api-access-4wqbk\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:57.011553 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:57.011519    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-maas-api-tls\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:57.014072 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:57.014042    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-maas-api-tls\") pod \"maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:57.224711 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:57.224671    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:57.596769 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:57.596738    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"]
Oct 30 19:01:57.597198 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:01:57.597175    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode678e320_130a_4cb1_a943_51c3467f266e.slice/crio-5651ffa145dbe7162bbae6ab86d52bb1d73c913164f94887584fae25ee0b76dd WatchSource:0}: Error finding container 5651ffa145dbe7162bbae6ab86d52bb1d73c913164f94887584fae25ee0b76dd: Status 404 returned error can't find the container with id 5651ffa145dbe7162bbae6ab86d52bb1d73c913164f94887584fae25ee0b76dd
Oct 30 19:01:57.664235 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:57.664194    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz" event={"ID":"e678e320-130a-4cb1-a943-51c3467f266e","Type":"ContainerStarted","Data":"5651ffa145dbe7162bbae6ab86d52bb1d73c913164f94887584fae25ee0b76dd"}
Oct 30 19:01:58.676060 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:58.676018    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz" event={"ID":"e678e320-130a-4cb1-a943-51c3467f266e","Type":"ContainerStarted","Data":"ed8ee7f8eb639c61f65fa42a139e12f23e2cec80f4aa05fe87138e94cfeffa30"}
Oct 30 19:01:58.676558 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:58.676256    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:01:58.820908 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:01:58.820803    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz" podStartSLOduration=1.9546932620000002 podStartE2EDuration="2.820789288s" totalImagesPullingTime="866.096026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:58.81783471 +0000 UTC m=+1513.928567994" watchObservedRunningTime="2026-10-30 19:01:58.820789288 +0000 UTC m=+1513.931522570"
Oct 30 19:02:04.688014 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:04.687981    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:02:11.398064 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:02:11.398022    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:02:11.482037 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:11.481987    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:02:20.427512 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.427478    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"]
Oct 30 19:02:20.431353 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.431334    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.484127 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.484095    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"]
Oct 30 19:02:20.526003 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.525962    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-15046cf2-serving-cert\""
Oct 30 19:02:20.585848 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.583499    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqsv\" (UniqueName: \"kubernetes.io/projected/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-kube-api-access-dfqsv\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.585848 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.583566    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-maas-api-tls\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.585848 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.583789    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-openshift-service-ca\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.585848 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.583876    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-metrics-certs\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.691654 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.685655    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-metrics-certs\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.691654 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.685854    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqsv\" (UniqueName: \"kubernetes.io/projected/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-kube-api-access-dfqsv\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.691654 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.685916    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-maas-api-tls\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.691654 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.686136    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-openshift-service-ca\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.691654 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.686967    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-openshift-service-ca\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.700474 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:02:20.700437    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-serving-cert: secret "maas-api-e2e-ait-15046cf2-serving-cert" not found
Oct 30 19:02:20.700835 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:02:20.700804    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-maas-api-tls podName:ac3e39c1-cb8d-4704-aebc-1ae94c3c9094 nodeName:}" failed. No retries permitted until 2026-10-30 19:02:21.200760529 +0000 UTC m=+1536.311493817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-maas-api-tls") pod "maas-api-e2e-ait-15046cf2-95cbcc757-7brzq" (UID: "ac3e39c1-cb8d-4704-aebc-1ae94c3c9094") : secret "maas-api-e2e-ait-15046cf2-serving-cert" not found
Oct 30 19:02:20.731729 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.731694    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-metrics-certs\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:20.756086 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:20.756052    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqsv\" (UniqueName: \"kubernetes.io/projected/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-kube-api-access-dfqsv\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:21.224695 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:21.224656    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-maas-api-tls\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:21.227069 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:21.227042    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-maas-api-tls\") pod \"maas-api-e2e-ait-15046cf2-95cbcc757-7brzq\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:21.348228 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:21.348193    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:21.716562 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:02:21.716527    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3e39c1_cb8d_4704_aebc_1ae94c3c9094.slice/crio-90b9cf1d2451ad3d01044a97fda2c56f2747d55dabb2c4e3abda3e7f2211ac6d WatchSource:0}: Error finding container 90b9cf1d2451ad3d01044a97fda2c56f2747d55dabb2c4e3abda3e7f2211ac6d: Status 404 returned error can't find the container with id 90b9cf1d2451ad3d01044a97fda2c56f2747d55dabb2c4e3abda3e7f2211ac6d
Oct 30 19:02:21.717005 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:21.716611    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"]
Oct 30 19:02:21.914416 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:21.914381    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq" event={"ID":"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094","Type":"ContainerStarted","Data":"90b9cf1d2451ad3d01044a97fda2c56f2747d55dabb2c4e3abda3e7f2211ac6d"}
Oct 30 19:02:23.079931 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:23.079886    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq" event={"ID":"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094","Type":"ContainerStarted","Data":"7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb"}
Oct 30 19:02:23.080526 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:23.080500    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:23.356013 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:23.354571    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq" podStartSLOduration=2.473542792 podStartE2EDuration="3.354545976s" totalImagesPullingTime="881.003184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:23.295356856 +0000 UTC m=+1538.406090143" watchObservedRunningTime="2026-10-30 19:02:23.354545976 +0000 UTC m=+1538.465279261"
Oct 30 19:02:29.141181 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:29.141154    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:02:41.254374 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:02:41.254329    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:02:43.570277 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:02:43.570242    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:03:11.260138 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:03:11.260042    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:03:12.232305 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.232268    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"]
Oct 30 19:03:12.236072 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.236052    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.268887 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.268851    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.269247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.268893    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5514f78-b153-49dc-9d09-9ba5e104ce78-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.269247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.268916    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.269247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.269037    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.269247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.269202    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.269247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.269241    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.269496 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.269451    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.269569 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.269548    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.269653 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.269638    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wlj\" (UniqueName: \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-kube-api-access-p2wlj\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.279099 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.279074    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"]
Oct 30 19:03:12.283146 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.283126    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.329428 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.329391    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"]
Oct 30 19:03:12.371437 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.371396    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.371437 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.371433    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.371722 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.371516    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.371722 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.371538    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.371839 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.371736    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.371839 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.371782    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.371950 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.371847    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.371950 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.371914    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.371950 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.371940    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.372136 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.371946    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.372136 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372057    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.372249 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372146    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.372249 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372176    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zzcp\" (UniqueName: \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-kube-api-access-6zzcp\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.372249 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372212    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.372406 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372267    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.372406 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372379    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wlj\" (UniqueName: \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-kube-api-access-p2wlj\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.372533 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372433    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.372533 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372474    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.372533 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372502    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.372533 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372510    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5514f78-b153-49dc-9d09-9ba5e104ce78-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.372765 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372650    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.372821 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.372787    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.373287 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.373260    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5514f78-b153-49dc-9d09-9ba5e104ce78-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.374772 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.374752    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.374892 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.374816    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.392835 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.392803    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"]
Oct 30 19:03:12.426086 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.426057    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.426912 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.426893    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wlj\" (UniqueName: \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-kube-api-access-p2wlj\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.474067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474033    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474233 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474075    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474233 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474095    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474233 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474212    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474358 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474300    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474385    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zzcp\" (UniqueName: \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-kube-api-access-6zzcp\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474448 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474437    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474505 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474447    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474559 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474537    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474559 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474542    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474691 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474662    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474787 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474743    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474932 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474915    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.474973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.474938    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.477061 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.477030    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.477662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.477639    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.528854 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.528783    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.528982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.528904    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zzcp\" (UniqueName: \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-kube-api-access-6zzcp\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.552500 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.552469    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:12.597954 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.597922    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:12.979529 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:12.979494    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"]
Oct 30 19:03:13.016913 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:03:13.016888    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5514f78_b153_49dc_9d09_9ba5e104ce78.slice/crio-08b237cb9dd27cc4901eb7cb3e32bc180b44e3192e1d5f62b570f162f6488faa WatchSource:0}: Error finding container 08b237cb9dd27cc4901eb7cb3e32bc180b44e3192e1d5f62b570f162f6488faa: Status 404 returned error can't find the container with id 08b237cb9dd27cc4901eb7cb3e32bc180b44e3192e1d5f62b570f162f6488faa
Oct 30 19:03:13.019412 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.019377    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:13.019522 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.019456    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:13.019522 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.019501    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:13.030843 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.030819    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"]
Oct 30 19:03:13.032449 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:03:13.032421    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e5190c_f468_4fb9_8572_3e7ff3f7f2cb.slice/crio-043cfe463f92b8f4e92cd47dc1ab7c6d45fe5b9901dd63742f16eea45b80224a WatchSource:0}: Error finding container 043cfe463f92b8f4e92cd47dc1ab7c6d45fe5b9901dd63742f16eea45b80224a: Status 404 returned error can't find the container with id 043cfe463f92b8f4e92cd47dc1ab7c6d45fe5b9901dd63742f16eea45b80224a
Oct 30 19:03:13.050608 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.035281    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:13.050608 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.035385    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:13.050608 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.046675    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:13.703659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.703619    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" event={"ID":"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb","Type":"ContainerStarted","Data":"3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072"}
Oct 30 19:03:13.704114 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.703673    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" event={"ID":"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb","Type":"ContainerStarted","Data":"043cfe463f92b8f4e92cd47dc1ab7c6d45fe5b9901dd63742f16eea45b80224a"}
Oct 30 19:03:13.705625 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.705561    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" event={"ID":"b5514f78-b153-49dc-9d09-9ba5e104ce78","Type":"ContainerStarted","Data":"e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6"}
Oct 30 19:03:13.705815 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.705641    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" event={"ID":"b5514f78-b153-49dc-9d09-9ba5e104ce78","Type":"ContainerStarted","Data":"08b237cb9dd27cc4901eb7cb3e32bc180b44e3192e1d5f62b570f162f6488faa"}
Oct 30 19:03:13.820609 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.820530    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" podStartSLOduration=1.820514705 podStartE2EDuration="1.820514705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:13.817296002 +0000 UTC m=+1588.928029286" watchObservedRunningTime="2026-10-30 19:03:13.820514705 +0000 UTC m=+1588.931247987"
Oct 30 19:03:13.941638 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:13.941570    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" podStartSLOduration=1.941556116 podStartE2EDuration="1.941556116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:13.939427628 +0000 UTC m=+1589.050160912" watchObservedRunningTime="2026-10-30 19:03:13.941556116 +0000 UTC m=+1589.052289481"
Oct 30 19:03:14.568606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:14.563943    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:14.568606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:14.566368    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:03:14.568606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:14.566517    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:03:14.599862 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:14.599764    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:14.600316 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:14.600294    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:03:14.600466 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:14.600445    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:03:15.596607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:15.591042    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:03:15.596607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:15.591110    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:03:15.601632 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:15.599649    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:03:15.601632 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:15.599704    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:03:16.562628 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:16.557102    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:03:16.562628 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:16.557166    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:03:16.601622 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:16.601565    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:03:16.603032 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:16.602995    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:03:17.554985 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:17.554944    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:03:17.555244 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:17.555215    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:03:17.598523 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:17.598485    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:03:17.598751 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:17.598547    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:03:18.618663 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:18.594163    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:03:18.626240 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:18.626198    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:03:18.626982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:18.621213    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:03:18.631606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:18.631469    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:03:19.557096 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:19.557053    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:03:19.557295 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:19.557137    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:03:19.612630 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:19.612592    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:03:19.612881 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:19.612858    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:03:20.553073 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:20.553037    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:03:20.553573 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:20.553101    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 30 19:03:20.598666 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:20.598624    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:03:20.598840 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:20.598694    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 19:03:21.557044 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:21.557015    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:21.557520 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:21.557499    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:21.558220 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:21.558200    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:03:21.651959 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:21.650816    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:21.652543 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:21.652400    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:21.654452 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:21.654389    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:03:23.554057 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.554023    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"]
Oct 30 19:03:23.557824 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.557806    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.566422 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.566394    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"]
Oct 30 19:03:23.607757 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.607728    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-06203ab6-serving-cert\""
Oct 30 19:03:23.633433 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.633394    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665346bc-6b95-4ba3-9b94-34c4cd020b1b-openshift-service-ca\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.633632 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.633502    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-metrics-certs\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.633787 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.633770    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpz5z\" (UniqueName: \"kubernetes.io/projected/665346bc-6b95-4ba3-9b94-34c4cd020b1b-kube-api-access-xpz5z\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.633934 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.633889    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-maas-api-tls\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.755544 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.755488    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpz5z\" (UniqueName: \"kubernetes.io/projected/665346bc-6b95-4ba3-9b94-34c4cd020b1b-kube-api-access-xpz5z\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.755861 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.755665    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-maas-api-tls\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.755861 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.755723    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665346bc-6b95-4ba3-9b94-34c4cd020b1b-openshift-service-ca\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.755861 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.755783    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-metrics-certs\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.779328 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:03:23.759714    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-serving-cert: secret "maas-api-e2e-mt-06203ab6-serving-cert" not found
Oct 30 19:03:23.779328 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:03:23.759815    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-maas-api-tls podName:665346bc-6b95-4ba3-9b94-34c4cd020b1b nodeName:}" failed. No retries permitted until 2026-10-30 19:03:24.259791033 +0000 UTC m=+1599.370524308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-maas-api-tls") pod "maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb" (UID: "665346bc-6b95-4ba3-9b94-34c4cd020b1b") : secret "maas-api-e2e-mt-06203ab6-serving-cert" not found
Oct 30 19:03:23.779328 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.761078    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665346bc-6b95-4ba3-9b94-34c4cd020b1b-openshift-service-ca\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.781181 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.781142    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-metrics-certs\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:23.817337 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:23.817219    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpz5z\" (UniqueName: \"kubernetes.io/projected/665346bc-6b95-4ba3-9b94-34c4cd020b1b-kube-api-access-xpz5z\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:24.270173 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:24.270135    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-maas-api-tls\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:24.272573 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:24.272546    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-maas-api-tls\") pod \"maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:24.477031 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:24.476998    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:24.850595 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:24.850557    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"]
Oct 30 19:03:24.850912 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:03:24.850666    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665346bc_6b95_4ba3_9b94_34c4cd020b1b.slice/crio-a24723e312b3ad5c54d040b22e671d435576de94bd5d4c5e7ea7475ac4c3c4cc WatchSource:0}: Error finding container a24723e312b3ad5c54d040b22e671d435576de94bd5d4c5e7ea7475ac4c3c4cc: Status 404 returned error can't find the container with id a24723e312b3ad5c54d040b22e671d435576de94bd5d4c5e7ea7475ac4c3c4cc
Oct 30 19:03:25.025942 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:25.025891    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb" event={"ID":"665346bc-6b95-4ba3-9b94-34c4cd020b1b","Type":"ContainerStarted","Data":"a24723e312b3ad5c54d040b22e671d435576de94bd5d4c5e7ea7475ac4c3c4cc"}
Oct 30 19:03:26.056681 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:26.055179    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:03:26.079645 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:26.079607    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb" event={"ID":"665346bc-6b95-4ba3-9b94-34c4cd020b1b","Type":"ContainerStarted","Data":"220090984a6568149a9ba3f8f4fd2323d08bb4b21adc754d887d85c822729046"}
Oct 30 19:03:26.080975 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:26.080950    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:26.191208 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:26.191150    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb" podStartSLOduration=2.386460184 podStartE2EDuration="3.191135012s" totalImagesPullingTime="804.674828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:26.189263086 +0000 UTC m=+1601.299996369" watchObservedRunningTime="2026-10-30 19:03:26.191135012 +0000 UTC m=+1601.301868295"
Oct 30 19:03:33.126642 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:33.126606    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:03:38.767434 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:38.767398    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"]
Oct 30 19:03:41.254736 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:03:41.254698    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:03:43.768444 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:43.768388    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn" podUID="f6a49b9e-9650-44b1-88c2-1ab6cc59289f" containerName="payload-pre-processing" containerID="cri-o://21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4" gracePeriod=25
Oct 30 19:03:43.969292 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:43.969253    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"]
Oct 30 19:03:43.969546 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:43.969516    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww" podUID="b8c33062-7b1f-46ef-941e-43475212a5ee" containerName="maas-api" containerID="cri-o://648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f" gracePeriod=30
Oct 30 19:03:44.122849 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.122805    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:03:44.251618 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.231077    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-plugins-config-volume\") pod \"f6a49b9e-9650-44b1-88c2-1ab6cc59289f\" (UID: \"f6a49b9e-9650-44b1-88c2-1ab6cc59289f\") "
Oct 30 19:03:44.251618 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.231162    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-kube-api-access-w2nmf\" (UniqueName: \"kubernetes.io/projected/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-kube-api-access-w2nmf\") pod \"f6a49b9e-9650-44b1-88c2-1ab6cc59289f\" (UID: \"f6a49b9e-9650-44b1-88c2-1ab6cc59289f\") "
Oct 30 19:03:44.467242 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.391845    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-plugins-config-volume" pod "f6a49b9e-9650-44b1-88c2-1ab6cc59289f" (UID: "f6a49b9e-9650-44b1-88c2-1ab6cc59289f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:44.467242 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.423036    4391 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-plugins-config-volume\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:44.569796 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.569764    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:03:44.627622 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.626448    4391 generic.go:356] "Generic (PLEG): container finished" podID="b8c33062-7b1f-46ef-941e-43475212a5ee" containerID="648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f" exitCode=0
Oct 30 19:03:44.627622 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.626656    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww" event={"ID":"b8c33062-7b1f-46ef-941e-43475212a5ee","Type":"ContainerDied","Data":"648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f"}
Oct 30 19:03:44.627622 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.626706    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww" event={"ID":"b8c33062-7b1f-46ef-941e-43475212a5ee","Type":"ContainerDied","Data":"9f199b3eedc9d7a6edf6a15f578ba8b3a3515ea72720b31a78c763479a6fc3ee"}
Oct 30 19:03:44.627622 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.626734    4391 scope.go:122] "RemoveContainer" containerID="648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f"
Oct 30 19:03:44.627622 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.626906    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"
Oct 30 19:03:44.679870 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.679838    4391 scope.go:122] "RemoveContainer" containerID="648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f"
Oct 30 19:03:44.680416 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.680385    4391 generic.go:356] "Generic (PLEG): container finished" podID="f6a49b9e-9650-44b1-88c2-1ab6cc59289f" containerID="21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4" exitCode=0
Oct 30 19:03:44.685452 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.685429    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"
Oct 30 19:03:44.697082 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.697046    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn" event={"ID":"f6a49b9e-9650-44b1-88c2-1ab6cc59289f","Type":"ContainerDied","Data":"21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4"}
Oct 30 19:03:44.701410 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.701373    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn" event={"ID":"f6a49b9e-9650-44b1-88c2-1ab6cc59289f","Type":"ContainerDied","Data":"e7fff72bce925f1d7134bd9a39d10aaa3286a9dd14c6eb720b371d4d04978ecd"}
Oct 30 19:03:44.701614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.696822    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-kube-api-access-w2nmf" pod "f6a49b9e-9650-44b1-88c2-1ab6cc59289f" (UID: "f6a49b9e-9650-44b1-88c2-1ab6cc59289f"). InnerVolumeSpecName "kube-api-access-w2nmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:44.701724 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:03:44.696251    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f\": container with ID starting with 648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f not found: ID does not exist" containerID="648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f"
Oct 30 19:03:44.701826 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.701805    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f"} err="failed to get container status \"648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f\": rpc error: code = NotFound desc = could not find container \"648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f\": container with ID starting with 648de88765f527a1f2260bfacb68fb14ec1bd46fa6cc983441959e672f7b5a0f not found: ID does not exist"
Oct 30 19:03:44.701943 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.701929    4391 scope.go:122] "RemoveContainer" containerID="21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4"
Oct 30 19:03:44.732678 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.732653    4391 scope.go:122] "RemoveContainer" containerID="21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4"
Oct 30 19:03:44.733102 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:03:44.733072    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4\": container with ID starting with 21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4 not found: ID does not exist" containerID="21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4"
Oct 30 19:03:44.733210 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.733115    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4"} err="failed to get container status \"21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4\": rpc error: code = NotFound desc = could not find container \"21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4\": container with ID starting with 21183b2ad0f71b0a3f9144dcadfccbedccbcd32c9f8345272533f0b68e77e6b4 not found: ID does not exist"
Oct 30 19:03:44.750546 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.750518    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-metrics-certs\") pod \"b8c33062-7b1f-46ef-941e-43475212a5ee\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") "
Oct 30 19:03:44.750694 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.750606    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-maas-api-tls\") pod \"b8c33062-7b1f-46ef-941e-43475212a5ee\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") "
Oct 30 19:03:44.750754 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.750736    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c33062-7b1f-46ef-941e-43475212a5ee-kube-api-access-phhdr\" (UniqueName: \"kubernetes.io/projected/b8c33062-7b1f-46ef-941e-43475212a5ee-kube-api-access-phhdr\") pod \"b8c33062-7b1f-46ef-941e-43475212a5ee\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") "
Oct 30 19:03:44.750891 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.750873    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c33062-7b1f-46ef-941e-43475212a5ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c33062-7b1f-46ef-941e-43475212a5ee-openshift-service-ca\") pod \"b8c33062-7b1f-46ef-941e-43475212a5ee\" (UID: \"b8c33062-7b1f-46ef-941e-43475212a5ee\") "
Oct 30 19:03:44.751329 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.751295    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c33062-7b1f-46ef-941e-43475212a5ee-openshift-service-ca" pod "b8c33062-7b1f-46ef-941e-43475212a5ee" (UID: "b8c33062-7b1f-46ef-941e-43475212a5ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:44.751681 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.751661    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2nmf\" (UniqueName: \"kubernetes.io/projected/f6a49b9e-9650-44b1-88c2-1ab6cc59289f-kube-api-access-w2nmf\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:44.751758 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.751688    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c33062-7b1f-46ef-941e-43475212a5ee-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:44.753175 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.753140    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-metrics-certs" pod "b8c33062-7b1f-46ef-941e-43475212a5ee" (UID: "b8c33062-7b1f-46ef-941e-43475212a5ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:44.753506 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.753477    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c33062-7b1f-46ef-941e-43475212a5ee-kube-api-access-phhdr" pod "b8c33062-7b1f-46ef-941e-43475212a5ee" (UID: "b8c33062-7b1f-46ef-941e-43475212a5ee"). InnerVolumeSpecName "kube-api-access-phhdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:44.753631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.753528    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-maas-api-tls" pod "b8c33062-7b1f-46ef-941e-43475212a5ee" (UID: "b8c33062-7b1f-46ef-941e-43475212a5ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:44.852977 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.852941    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:44.852977 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.852974    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c33062-7b1f-46ef-941e-43475212a5ee-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:44.852977 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:44.852984    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phhdr\" (UniqueName: \"kubernetes.io/projected/b8c33062-7b1f-46ef-941e-43475212a5ee-kube-api-access-phhdr\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:45.118550 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:45.118518    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"]
Oct 30 19:03:45.122865 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:45.122835    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-948bc488-6b65bd4db9-wlpww"]
Oct 30 19:03:45.226733 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:45.226694    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"]
Oct 30 19:03:45.230835 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:45.230806    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15046cf2-85d9577df5-gz4mn"]
Oct 30 19:03:47.008181 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:47.008147    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c33062-7b1f-46ef-941e-43475212a5ee" path="/var/lib/kubelet/pods/b8c33062-7b1f-46ef-941e-43475212a5ee/volumes"
Oct 30 19:03:47.008665 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:47.008649    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a49b9e-9650-44b1-88c2-1ab6cc59289f" path="/var/lib/kubelet/pods/f6a49b9e-9650-44b1-88c2-1ab6cc59289f/volumes"
Oct 30 19:03:50.227617 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.227564    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"]
Oct 30 19:03:50.228049 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.227809    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz" podUID="e678e320-130a-4cb1-a943-51c3467f266e" containerName="maas-api" containerID="cri-o://ed8ee7f8eb639c61f65fa42a139e12f23e2cec80f4aa05fe87138e94cfeffa30" gracePeriod=30
Oct 30 19:03:50.865126 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.865077    4391 generic.go:356] "Generic (PLEG): container finished" podID="e678e320-130a-4cb1-a943-51c3467f266e" containerID="ed8ee7f8eb639c61f65fa42a139e12f23e2cec80f4aa05fe87138e94cfeffa30" exitCode=0
Oct 30 19:03:50.867083 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.865521    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz" event={"ID":"e678e320-130a-4cb1-a943-51c3467f266e","Type":"ContainerDied","Data":"ed8ee7f8eb639c61f65fa42a139e12f23e2cec80f4aa05fe87138e94cfeffa30"}
Oct 30 19:03:50.867292 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.867272    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz" event={"ID":"e678e320-130a-4cb1-a943-51c3467f266e","Type":"ContainerDied","Data":"5651ffa145dbe7162bbae6ab86d52bb1d73c913164f94887584fae25ee0b76dd"}
Oct 30 19:03:50.867395 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.867381    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5651ffa145dbe7162bbae6ab86d52bb1d73c913164f94887584fae25ee0b76dd"
Oct 30 19:03:50.873171 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.873139    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:03:50.966372 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.964572    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e678e320-130a-4cb1-a943-51c3467f266e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e678e320-130a-4cb1-a943-51c3467f266e-openshift-service-ca\") pod \"e678e320-130a-4cb1-a943-51c3467f266e\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") "
Oct 30 19:03:50.966372 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.964698    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e678e320-130a-4cb1-a943-51c3467f266e-kube-api-access-4wqbk\" (UniqueName: \"kubernetes.io/projected/e678e320-130a-4cb1-a943-51c3467f266e-kube-api-access-4wqbk\") pod \"e678e320-130a-4cb1-a943-51c3467f266e\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") "
Oct 30 19:03:50.966372 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.964870    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-metrics-certs\") pod \"e678e320-130a-4cb1-a943-51c3467f266e\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") "
Oct 30 19:03:50.966372 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.965050    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-maas-api-tls\") pod \"e678e320-130a-4cb1-a943-51c3467f266e\" (UID: \"e678e320-130a-4cb1-a943-51c3467f266e\") "
Oct 30 19:03:50.975630 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.974957    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e678e320-130a-4cb1-a943-51c3467f266e-kube-api-access-4wqbk" pod "e678e320-130a-4cb1-a943-51c3467f266e" (UID: "e678e320-130a-4cb1-a943-51c3467f266e"). InnerVolumeSpecName "kube-api-access-4wqbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:50.975630 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.975303    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e678e320-130a-4cb1-a943-51c3467f266e-openshift-service-ca" pod "e678e320-130a-4cb1-a943-51c3467f266e" (UID: "e678e320-130a-4cb1-a943-51c3467f266e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:50.979641 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.978415    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-metrics-certs" pod "e678e320-130a-4cb1-a943-51c3467f266e" (UID: "e678e320-130a-4cb1-a943-51c3467f266e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:50.995603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:50.995479    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-maas-api-tls" pod "e678e320-130a-4cb1-a943-51c3467f266e" (UID: "e678e320-130a-4cb1-a943-51c3467f266e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:51.096669 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:51.096541    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.096669 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:51.096638    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e678e320-130a-4cb1-a943-51c3467f266e-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.096669 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:51.096656    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e678e320-130a-4cb1-a943-51c3467f266e-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.096669 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:51.096670    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wqbk\" (UniqueName: \"kubernetes.io/projected/e678e320-130a-4cb1-a943-51c3467f266e-kube-api-access-4wqbk\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.873929 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:51.873891    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"
Oct 30 19:03:52.075375 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:52.075326    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"]
Oct 30 19:03:52.121224 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:52.121189    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-85b6ee7b-7449f5765b-prddz"]
Oct 30 19:03:53.007810 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:53.007770    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e678e320-130a-4cb1-a943-51c3467f266e" path="/var/lib/kubelet/pods/e678e320-130a-4cb1-a943-51c3467f266e/volumes"
Oct 30 19:03:54.907340 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.907302    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"]
Oct 30 19:03:54.912009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.908855    4391 cpu_manager.go:414] "containerMap: removing container" podUID="e678e320-130a-4cb1-a943-51c3467f266e" containerName="maas-api"
Oct 30 19:03:54.912009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.908886    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e678e320-130a-4cb1-a943-51c3467f266e" containerName="maas-api"
Oct 30 19:03:54.912009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.908941    4391 cpu_manager.go:414] "containerMap: removing container" podUID="f6a49b9e-9650-44b1-88c2-1ab6cc59289f" containerName="payload-pre-processing"
Oct 30 19:03:54.912009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.908951    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a49b9e-9650-44b1-88c2-1ab6cc59289f" containerName="payload-pre-processing"
Oct 30 19:03:54.912009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.908993    4391 cpu_manager.go:414] "containerMap: removing container" podUID="b8c33062-7b1f-46ef-941e-43475212a5ee" containerName="maas-api"
Oct 30 19:03:54.912009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.909003    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c33062-7b1f-46ef-941e-43475212a5ee" containerName="maas-api"
Oct 30 19:03:54.912009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.909182    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="e678e320-130a-4cb1-a943-51c3467f266e" containerName="maas-api"
Oct 30 19:03:54.912009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.909213    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a49b9e-9650-44b1-88c2-1ab6cc59289f" containerName="payload-pre-processing"
Oct 30 19:03:54.912009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.909230    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c33062-7b1f-46ef-941e-43475212a5ee" containerName="maas-api"
Oct 30 19:03:54.922612 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.920496    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:03:55.001607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:54.994092    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"]
Oct 30 19:03:55.032608 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.016286    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4cef683f-dockercfg-5pw4h\""
Oct 30 19:03:55.032608 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.016986    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4cef683f\""
Oct 30 19:03:55.136616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.131160    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f95989-1ca6-443a-bf64-53eb4654f6b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr\" (UID: \"e4f95989-1ca6-443a-bf64-53eb4654f6b4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:03:55.136616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.131266    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffcw\" (UniqueName: \"kubernetes.io/projected/e4f95989-1ca6-443a-bf64-53eb4654f6b4-kube-api-access-nffcw\") pod \"payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr\" (UID: \"e4f95989-1ca6-443a-bf64-53eb4654f6b4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:03:55.235653 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.234773    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffcw\" (UniqueName: \"kubernetes.io/projected/e4f95989-1ca6-443a-bf64-53eb4654f6b4-kube-api-access-nffcw\") pod \"payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr\" (UID: \"e4f95989-1ca6-443a-bf64-53eb4654f6b4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:03:55.235653 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.235219    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f95989-1ca6-443a-bf64-53eb4654f6b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr\" (UID: \"e4f95989-1ca6-443a-bf64-53eb4654f6b4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:03:55.257604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.236218    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f95989-1ca6-443a-bf64-53eb4654f6b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr\" (UID: \"e4f95989-1ca6-443a-bf64-53eb4654f6b4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:03:55.257604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.244223    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"]
Oct 30 19:03:55.275647 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.263810    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"]
Oct 30 19:03:55.275647 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.263949    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:03:55.311609 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.311508    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffcw\" (UniqueName: \"kubernetes.io/projected/e4f95989-1ca6-443a-bf64-53eb4654f6b4-kube-api-access-nffcw\") pod \"payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr\" (UID: \"e4f95989-1ca6-443a-bf64-53eb4654f6b4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:03:55.355348 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.347101    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-plugins-config-volume\") pod \"payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw\" (UID: \"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33\") " pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:03:55.355348 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.347255    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqkq\" (UniqueName: \"kubernetes.io/projected/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-kube-api-access-dfqkq\") pod \"payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw\" (UID: \"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33\") " pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:03:55.459550 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.455246    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-plugins-config-volume\") pod \"payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw\" (UID: \"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33\") " pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:03:55.459550 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.455393    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqkq\" (UniqueName: \"kubernetes.io/projected/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-kube-api-access-dfqkq\") pod \"payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw\" (UID: \"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33\") " pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:03:55.482691 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.479116    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-plugins-config-volume\") pod \"payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw\" (UID: \"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33\") " pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:03:55.520647 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.518991    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqkq\" (UniqueName: \"kubernetes.io/projected/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-kube-api-access-dfqkq\") pod \"payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw\" (UID: \"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33\") " pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:03:55.555663 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.555626    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:03:55.652816 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.652783    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:03:55.941498 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:55.941458    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"]
Oct 30 19:03:56.045757 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:56.045728    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"]
Oct 30 19:03:56.045873 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:03:56.045723    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e57ff5_bec2_4f3c_9ff8_fedfad75ed33.slice/crio-894e2b03fc5cfada31a6315efa8b89435bb652c62c67de8c43ab087968b251b5 WatchSource:0}: Error finding container 894e2b03fc5cfada31a6315efa8b89435bb652c62c67de8c43ab087968b251b5: Status 404 returned error can't find the container with id 894e2b03fc5cfada31a6315efa8b89435bb652c62c67de8c43ab087968b251b5
Oct 30 19:03:56.937634 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:56.937596    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr" event={"ID":"e4f95989-1ca6-443a-bf64-53eb4654f6b4","Type":"ContainerStarted","Data":"c875bf563b022b5fe416318c7b99c587ed7081f44c1f1f78509c0bae6974ad6c"}
Oct 30 19:03:56.938250 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:56.937668    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr" event={"ID":"e4f95989-1ca6-443a-bf64-53eb4654f6b4","Type":"ContainerStarted","Data":"7c62c18ebabde3cdfdd5adf45e8857f9fc43a63911028481e2a0c4260a8048aa"}
Oct 30 19:03:56.938250 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:56.937832    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:03:56.939895 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:56.939864    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw" event={"ID":"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33","Type":"ContainerStarted","Data":"9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e"}
Oct 30 19:03:56.940039 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:56.939917    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw" event={"ID":"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33","Type":"ContainerStarted","Data":"894e2b03fc5cfada31a6315efa8b89435bb652c62c67de8c43ab087968b251b5"}
Oct 30 19:03:56.940039 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:56.939962    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:03:57.053743 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:57.053695    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr" podStartSLOduration=3.05368114 podStartE2EDuration="3.05368114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:57.05270532 +0000 UTC m=+1632.163438603" watchObservedRunningTime="2026-10-30 19:03:57.05368114 +0000 UTC m=+1632.164414422"
Oct 30 19:03:57.164675 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:03:57.164615    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw" podStartSLOduration=3.1645998300000002 podStartE2EDuration="3.16459983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:57.159612525 +0000 UTC m=+1632.270345810" watchObservedRunningTime="2026-10-30 19:03:57.16459983 +0000 UTC m=+1632.275333114"
Oct 30 19:04:07.949967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:07.949935    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:04:07.950381 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:07.950002    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:04:11.256804 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:04:11.256761    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:04:11.933267 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:11.933228    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"]
Oct 30 19:04:11.937670 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:11.937652    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.034138 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.034106    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-3bd578-serving-cert\""
Oct 30 19:04:12.041152 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.041121    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"]
Oct 30 19:04:12.084091 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.084050    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdlhk\" (UniqueName: \"kubernetes.io/projected/79564581-842c-4551-8880-b31c78cd608e-kube-api-access-bdlhk\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.084275 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.084179    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.084404 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.084382    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79564581-842c-4551-8880-b31c78cd608e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.084477 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.084458    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.229658 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.212476    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79564581-842c-4551-8880-b31c78cd608e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.236080 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.231486    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.236372 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:04:12.235805    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-serving-cert: secret "maas-api-e2e-shared-a-w1-3bd578-serving-cert" not found
Oct 30 19:04:12.236486 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:04:12.236455    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-maas-api-tls podName:79564581-842c-4551-8880-b31c78cd608e nodeName:}" failed. No retries permitted until 2026-10-30 19:04:12.736426995 +0000 UTC m=+1647.847160264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-maas-api-tls") pod "maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj" (UID: "79564581-842c-4551-8880-b31c78cd608e") : secret "maas-api-e2e-shared-a-w1-3bd578-serving-cert" not found
Oct 30 19:04:12.236685 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.236544    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlhk\" (UniqueName: \"kubernetes.io/projected/79564581-842c-4551-8880-b31c78cd608e-kube-api-access-bdlhk\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.236685 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.236646    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.236919 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.230926    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79564581-842c-4551-8880-b31c78cd608e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.240108 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.240086    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.303066 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.299785    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlhk\" (UniqueName: \"kubernetes.io/projected/79564581-842c-4551-8880-b31c78cd608e-kube-api-access-bdlhk\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.748095 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.748058    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.750481 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.750452    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:12.853372 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:12.853333    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:13.225528 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:04:13.225495    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79564581_842c_4551_8880_b31c78cd608e.slice/crio-7c49947b53560c175f4bad41df3b8132f1d63da6d2feae7123e88acea5b99dec WatchSource:0}: Error finding container 7c49947b53560c175f4bad41df3b8132f1d63da6d2feae7123e88acea5b99dec: Status 404 returned error can't find the container with id 7c49947b53560c175f4bad41df3b8132f1d63da6d2feae7123e88acea5b99dec
Oct 30 19:04:13.225677 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:13.225571    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"]
Oct 30 19:04:14.172028 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:14.171991    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj" event={"ID":"79564581-842c-4551-8880-b31c78cd608e","Type":"ContainerStarted","Data":"7c49947b53560c175f4bad41df3b8132f1d63da6d2feae7123e88acea5b99dec"}
Oct 30 19:04:15.182023 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:15.181985    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj" event={"ID":"79564581-842c-4551-8880-b31c78cd608e","Type":"ContainerStarted","Data":"b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97"}
Oct 30 19:04:15.182540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:15.182096    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:15.295234 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:15.295176    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj" podStartSLOduration=3.33963789 podStartE2EDuration="4.295161539s" totalImagesPullingTime="955.523649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:15.294530716 +0000 UTC m=+1650.405263993" watchObservedRunningTime="2026-10-30 19:04:15.295161539 +0000 UTC m=+1650.405894822"
Oct 30 19:04:16.411708 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:16.411675    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"]
Oct 30 19:04:16.412090 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:16.411891    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq" podUID="ac3e39c1-cb8d-4704-aebc-1ae94c3c9094" containerName="maas-api" containerID="cri-o://7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb" gracePeriod=30
Oct 30 19:04:16.912287 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:16.912251    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:04:17.110920 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.110279    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-kube-api-access-dfqsv\" (UniqueName: \"kubernetes.io/projected/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-kube-api-access-dfqsv\") pod \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") "
Oct 30 19:04:17.111145 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.111124    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-maas-api-tls\") pod \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") "
Oct 30 19:04:17.111807 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.111786    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-openshift-service-ca\") pod \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") "
Oct 30 19:04:17.112005 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.111989    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-metrics-certs\") pod \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\" (UID: \"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094\") "
Oct 30 19:04:17.112391 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.112362    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-openshift-service-ca" pod "ac3e39c1-cb8d-4704-aebc-1ae94c3c9094" (UID: "ac3e39c1-cb8d-4704-aebc-1ae94c3c9094"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:17.137414 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.137383    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:17.138930 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.138897    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-kube-api-access-dfqsv" pod "ac3e39c1-cb8d-4704-aebc-1ae94c3c9094" (UID: "ac3e39c1-cb8d-4704-aebc-1ae94c3c9094"). InnerVolumeSpecName "kube-api-access-dfqsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:17.139518 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.139490    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-maas-api-tls" pod "ac3e39c1-cb8d-4704-aebc-1ae94c3c9094" (UID: "ac3e39c1-cb8d-4704-aebc-1ae94c3c9094"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:17.141259 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.141224    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-metrics-certs" pod "ac3e39c1-cb8d-4704-aebc-1ae94c3c9094" (UID: "ac3e39c1-cb8d-4704-aebc-1ae94c3c9094"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:17.233722 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.232239    4391 generic.go:356] "Generic (PLEG): container finished" podID="ac3e39c1-cb8d-4704-aebc-1ae94c3c9094" containerID="7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb" exitCode=0
Oct 30 19:04:17.233722 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.232425    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"
Oct 30 19:04:17.233722 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.232500    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq" event={"ID":"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094","Type":"ContainerDied","Data":"7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb"}
Oct 30 19:04:17.233722 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.232550    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq" event={"ID":"ac3e39c1-cb8d-4704-aebc-1ae94c3c9094","Type":"ContainerDied","Data":"90b9cf1d2451ad3d01044a97fda2c56f2747d55dabb2c4e3abda3e7f2211ac6d"}
Oct 30 19:04:17.233722 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.232595    4391 scope.go:122] "RemoveContainer" containerID="7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb"
Oct 30 19:04:17.239663 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.239181    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqsv\" (UniqueName: \"kubernetes.io/projected/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-kube-api-access-dfqsv\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:17.239663 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.239215    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:17.239663 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.239229    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:17.264403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.264384    4391 scope.go:122] "RemoveContainer" containerID="7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb"
Oct 30 19:04:17.264971 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:04:17.264942    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb\": container with ID starting with 7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb not found: ID does not exist" containerID="7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb"
Oct 30 19:04:17.265063 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.264985    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb"} err="failed to get container status \"7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb\": rpc error: code = NotFound desc = could not find container \"7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb\": container with ID starting with 7f41416a99268f45dd2375efb638c32fe71c702aae31039de4e5d5a25773decb not found: ID does not exist"
Oct 30 19:04:17.416605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.415449    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"]
Oct 30 19:04:17.468601 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:17.459294    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15046cf2-95cbcc757-7brzq"]
Oct 30 19:04:19.066601 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:19.060213    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3e39c1-cb8d-4704-aebc-1ae94c3c9094" path="/var/lib/kubelet/pods/ac3e39c1-cb8d-4704-aebc-1ae94c3c9094/volumes"
Oct 30 19:04:21.210980 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:21.210953    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:04:33.362474 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.362400    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"]
Oct 30 19:04:33.363210 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.363190    4391 cpu_manager.go:414] "containerMap: removing container" podUID="ac3e39c1-cb8d-4704-aebc-1ae94c3c9094" containerName="maas-api"
Oct 30 19:04:33.363254 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.363212    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3e39c1-cb8d-4704-aebc-1ae94c3c9094" containerName="maas-api"
Oct 30 19:04:33.363328 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.363319    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3e39c1-cb8d-4704-aebc-1ae94c3c9094" containerName="maas-api"
Oct 30 19:04:33.368563 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.368544    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.420339 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.420304    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"]
Oct 30 19:04:33.460729 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.460698    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-8be536-serving-cert\""
Oct 30 19:04:33.475575 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.475542    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvc4w\" (UniqueName: \"kubernetes.io/projected/34b7732a-b154-4fdd-a7a9-e8f81b734a77-kube-api-access-tvc4w\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.475862 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.475832    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.475992 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.475865    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.476062 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.476012    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b7732a-b154-4fdd-a7a9-e8f81b734a77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.577081 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.577030    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.577081 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.577081    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.577336 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.577155    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b7732a-b154-4fdd-a7a9-e8f81b734a77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.577336 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:04:33.577173    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-serving-cert: secret "maas-api-e2e-shared-a-w6-8be536-serving-cert" not found
Oct 30 19:04:33.577336 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:04:33.577234    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-maas-api-tls podName:34b7732a-b154-4fdd-a7a9-e8f81b734a77 nodeName:}" failed. No retries permitted until 2026-10-30 19:04:34.07721906 +0000 UTC m=+1669.187952322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-maas-api-tls") pod "maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx" (UID: "34b7732a-b154-4fdd-a7a9-e8f81b734a77") : secret "maas-api-e2e-shared-a-w6-8be536-serving-cert" not found
Oct 30 19:04:33.577336 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.577261    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvc4w\" (UniqueName: \"kubernetes.io/projected/34b7732a-b154-4fdd-a7a9-e8f81b734a77-kube-api-access-tvc4w\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.577918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.577893    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b7732a-b154-4fdd-a7a9-e8f81b734a77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.579645 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.579629    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:33.669655 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:33.669547    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvc4w\" (UniqueName: \"kubernetes.io/projected/34b7732a-b154-4fdd-a7a9-e8f81b734a77-kube-api-access-tvc4w\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:34.094851 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:34.094760    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:34.097171 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:34.097147    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:34.283569 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:34.283534    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:34.708053 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:34.702554    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"]
Oct 30 19:04:34.899820 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:34.899781    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:04:35.377411 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:35.377372    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx" event={"ID":"34b7732a-b154-4fdd-a7a9-e8f81b734a77","Type":"ContainerStarted","Data":"d06aab6a44c8d054a280569f52fa85921f5adba08423d4d582a483ab62dbdce3"}
Oct 30 19:04:36.386386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:36.386340    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx" event={"ID":"34b7732a-b154-4fdd-a7a9-e8f81b734a77","Type":"ContainerStarted","Data":"799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0"}
Oct 30 19:04:36.386898 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:36.386454    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:04:36.555609 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:36.550800    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx" podStartSLOduration=2.618486105 podStartE2EDuration="3.55077846s" totalImagesPullingTime="932.292355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:36.500071172 +0000 UTC m=+1671.610804456" watchObservedRunningTime="2026-10-30 19:04:36.55077846 +0000 UTC m=+1671.661511747"
Oct 30 19:04:41.254073 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:04:41.254033    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:04:42.399511 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:04:42.399480    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:05:04.074842 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:04.074801    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"]
Oct 30 19:05:04.121138 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:04.121102    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad8aed8-nvsqh"]
Oct 30 19:05:05.009625 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:05.009573    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14a6ae5-8be4-4072-bb29-be96e93323b4" path="/var/lib/kubelet/pods/b14a6ae5-8be4-4072-bb29-be96e93323b4/volumes"
Oct 30 19:05:10.115239 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:10.115197    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"]
Oct 30 19:05:10.156348 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:10.156259    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74f04158-l6m74"]
Oct 30 19:05:11.079606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:11.079548    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69a8582-95df-4b49-a879-8b6f240c4478" path="/var/lib/kubelet/pods/b69a8582-95df-4b49-a879-8b6f240c4478/volumes"
Oct 30 19:05:11.254703 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:05:11.254665    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:05:26.731376 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:26.731341    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"]
Oct 30 19:05:26.731898 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:26.731555    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy" containerID="cri-o://331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38" gracePeriod=30
Oct 30 19:05:32.005546 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.005482    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:05:32.156461 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156425    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-certs\") pod \"71505255-630b-4cb2-bfda-5841a99f993b\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") "
Oct 30 19:05:32.156668 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156480    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71505255-630b-4cb2-bfda-5841a99f993b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71505255-630b-4cb2-bfda-5841a99f993b-istio-podinfo\") pod \"71505255-630b-4cb2-bfda-5841a99f993b\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") "
Oct 30 19:05:32.156668 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156605    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71505255-630b-4cb2-bfda-5841a99f993b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71505255-630b-4cb2-bfda-5841a99f993b-istiod-ca-cert\") pod \"71505255-630b-4cb2-bfda-5841a99f993b\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") "
Oct 30 19:05:32.156668 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156640    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-istio-token\" (UniqueName: \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-istio-token\") pod \"71505255-630b-4cb2-bfda-5841a99f993b\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") "
Oct 30 19:05:32.156855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156737    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-data\") pod \"71505255-630b-4cb2-bfda-5841a99f993b\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") "
Oct 30 19:05:32.156855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156775    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-certs" pod "71505255-630b-4cb2-bfda-5841a99f993b" (UID: "71505255-630b-4cb2-bfda-5841a99f993b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:32.156855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156792    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-socket\") pod \"71505255-630b-4cb2-bfda-5841a99f993b\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") "
Oct 30 19:05:32.156855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156824    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-credential-socket\") pod \"71505255-630b-4cb2-bfda-5841a99f993b\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") "
Oct 30 19:05:32.157050 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156874    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-envoy\") pod \"71505255-630b-4cb2-bfda-5841a99f993b\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") "
Oct 30 19:05:32.157050 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156908    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-kube-api-access-thdvb\" (UniqueName: \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-kube-api-access-thdvb\") pod \"71505255-630b-4cb2-bfda-5841a99f993b\" (UID: \"71505255-630b-4cb2-bfda-5841a99f993b\") "
Oct 30 19:05:32.157050 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156963    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-socket" pod "71505255-630b-4cb2-bfda-5841a99f993b" (UID: "71505255-630b-4cb2-bfda-5841a99f993b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:32.157050 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.156980    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-data" pod "71505255-630b-4cb2-bfda-5841a99f993b" (UID: "71505255-630b-4cb2-bfda-5841a99f993b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:32.157240 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.157125    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-credential-socket" pod "71505255-630b-4cb2-bfda-5841a99f993b" (UID: "71505255-630b-4cb2-bfda-5841a99f993b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:32.157322 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.157291    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71505255-630b-4cb2-bfda-5841a99f993b-istiod-ca-cert" pod "71505255-630b-4cb2-bfda-5841a99f993b" (UID: "71505255-630b-4cb2-bfda-5841a99f993b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:32.157832 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.157801    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.157832 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.157828    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71505255-630b-4cb2-bfda-5841a99f993b-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.158021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.157844    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.158021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.157858    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.158021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.157874    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.159744 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.159714    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-envoy" pod "71505255-630b-4cb2-bfda-5841a99f993b" (UID: "71505255-630b-4cb2-bfda-5841a99f993b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:32.159744 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.159731    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71505255-630b-4cb2-bfda-5841a99f993b-istio-podinfo" pod "71505255-630b-4cb2-bfda-5841a99f993b" (UID: "71505255-630b-4cb2-bfda-5841a99f993b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:05:32.159908 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.159755    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-kube-api-access-thdvb" pod "71505255-630b-4cb2-bfda-5841a99f993b" (UID: "71505255-630b-4cb2-bfda-5841a99f993b"). InnerVolumeSpecName "kube-api-access-thdvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:32.159962 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.159912    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-istio-token" pod "71505255-630b-4cb2-bfda-5841a99f993b" (UID: "71505255-630b-4cb2-bfda-5841a99f993b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:32.260341 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.260261    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71505255-630b-4cb2-bfda-5841a99f993b-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.260341 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.260302    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thdvb\" (UniqueName: \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-kube-api-access-thdvb\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.260341 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.260323    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71505255-630b-4cb2-bfda-5841a99f993b-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.260341 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.260336    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71505255-630b-4cb2-bfda-5841a99f993b-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.321949 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.321911    4391 generic.go:356] "Generic (PLEG): container finished" podID="71505255-630b-4cb2-bfda-5841a99f993b" containerID="331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38" exitCode=0
Oct 30 19:05:32.322118 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.321958    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"
Oct 30 19:05:32.322118 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.321992    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" event={"ID":"71505255-630b-4cb2-bfda-5841a99f993b","Type":"ContainerDied","Data":"331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38"}
Oct 30 19:05:32.322118 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.322049    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs" event={"ID":"71505255-630b-4cb2-bfda-5841a99f993b","Type":"ContainerDied","Data":"672ff40f5618d263e4749497ba5ea98511fbc3947ae57d24f9dc0736d209a059"}
Oct 30 19:05:32.322118 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.322073    4391 scope.go:122] "RemoveContainer" containerID="331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38"
Oct 30 19:05:32.333642 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.333625    4391 scope.go:122] "RemoveContainer" containerID="331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38"
Oct 30 19:05:32.333939 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:05:32.333920    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38\": container with ID starting with 331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38 not found: ID does not exist" containerID="331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38"
Oct 30 19:05:32.334042 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.333951    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38"} err="failed to get container status \"331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38\": rpc error: code = NotFound desc = could not find container \"331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38\": container with ID starting with 331ad00076e89e0b677ef27656c382b97430f023e10f50f2d4ad31668329cb38 not found: ID does not exist"
Oct 30 19:05:32.529605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.529508    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"]
Oct 30 19:05:32.548981 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.548947    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-v79rs"]
Oct 30 19:05:32.590558 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.590507    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"]
Oct 30 19:05:32.614799 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.609012    4391 cpu_manager.go:414] "containerMap: removing container" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy"
Oct 30 19:05:32.614799 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.609042    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy"
Oct 30 19:05:32.614799 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.609250    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="71505255-630b-4cb2-bfda-5841a99f993b" containerName="istio-proxy"
Oct 30 19:05:32.619014 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.618068    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:05:32.638610 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.636073    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"]
Oct 30 19:05:32.668996 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.668961    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-091e8f-dockercfg-sbmbh\""
Oct 30 19:05:32.671922 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.670985    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-091e8f\""
Oct 30 19:05:32.678776 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.678740    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3868727-94fa-41b9-bd83-a9ff0c537971-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr\" (UID: \"b3868727-94fa-41b9-bd83-a9ff0c537971\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:05:32.683950 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.683906    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmf9\" (UniqueName: \"kubernetes.io/projected/b3868727-94fa-41b9-bd83-a9ff0c537971-kube-api-access-lnmf9\") pod \"payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr\" (UID: \"b3868727-94fa-41b9-bd83-a9ff0c537971\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:05:32.795383 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.795287    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmf9\" (UniqueName: \"kubernetes.io/projected/b3868727-94fa-41b9-bd83-a9ff0c537971-kube-api-access-lnmf9\") pod \"payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr\" (UID: \"b3868727-94fa-41b9-bd83-a9ff0c537971\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:05:32.796297 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.796261    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3868727-94fa-41b9-bd83-a9ff0c537971-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr\" (UID: \"b3868727-94fa-41b9-bd83-a9ff0c537971\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:05:32.805602 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.797529    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3868727-94fa-41b9-bd83-a9ff0c537971-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr\" (UID: \"b3868727-94fa-41b9-bd83-a9ff0c537971\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:05:32.911464 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.887008    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmf9\" (UniqueName: \"kubernetes.io/projected/b3868727-94fa-41b9-bd83-a9ff0c537971-kube-api-access-lnmf9\") pod \"payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr\" (UID: \"b3868727-94fa-41b9-bd83-a9ff0c537971\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:05:32.963689 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:32.963649    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:05:33.016785 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:33.016739    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71505255-630b-4cb2-bfda-5841a99f993b" path="/var/lib/kubelet/pods/71505255-630b-4cb2-bfda-5841a99f993b/volumes"
Oct 30 19:05:33.360191 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:33.360153    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"]
Oct 30 19:05:33.361158 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:05:33.361133    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3868727_94fa_41b9_bd83_a9ff0c537971.slice/crio-010159b71fb8c0c40fa920a8b4401fed348f64807794e8966d83954b0d204cbe WatchSource:0}: Error finding container 010159b71fb8c0c40fa920a8b4401fed348f64807794e8966d83954b0d204cbe: Status 404 returned error can't find the container with id 010159b71fb8c0c40fa920a8b4401fed348f64807794e8966d83954b0d204cbe
Oct 30 19:05:34.338329 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:34.338286    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr" event={"ID":"b3868727-94fa-41b9-bd83-a9ff0c537971","Type":"ContainerStarted","Data":"03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc"}
Oct 30 19:05:34.338811 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:34.338340    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr" event={"ID":"b3868727-94fa-41b9-bd83-a9ff0c537971","Type":"ContainerStarted","Data":"010159b71fb8c0c40fa920a8b4401fed348f64807794e8966d83954b0d204cbe"}
Oct 30 19:05:34.338811 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:34.338493    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:05:34.455574 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:34.455515    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr" podStartSLOduration=2.45550247 podStartE2EDuration="2.45550247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:34.451465246 +0000 UTC m=+1729.562198529" watchObservedRunningTime="2026-10-30 19:05:34.45550247 +0000 UTC m=+1729.566235757"
Oct 30 19:05:38.052992 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:38.052954    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"]
Oct 30 19:05:38.099270 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:38.099232    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87003c5d-gccvk"]
Oct 30 19:05:39.007183 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:39.007150    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e" path="/var/lib/kubelet/pods/c7fac5dc-9c53-4bda-a3e6-a1da7809fd9e/volumes"
Oct 30 19:05:41.254474 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:05:41.254435    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:05:42.751685 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:42.751642    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:05:45.349662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:05:45.349623    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:06:06.566479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.566445    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"]
Oct 30 19:06:06.571447 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.571426    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.624930 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.624896    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"]
Oct 30 19:06:06.666646 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.666600    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-314bd6fa-serving-cert\""
Oct 30 19:06:06.738335 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.738302    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-metrics-certs\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.738512 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.738458    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/568b301b-486f-4cf6-8d4a-26598ba2fef0-openshift-service-ca\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.738611 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.738595    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tll6v\" (UniqueName: \"kubernetes.io/projected/568b301b-486f-4cf6-8d4a-26598ba2fef0-kube-api-access-tll6v\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.738662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.738639    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-maas-api-tls\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.840856 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.840762    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/568b301b-486f-4cf6-8d4a-26598ba2fef0-openshift-service-ca\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.841032 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.840914    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tll6v\" (UniqueName: \"kubernetes.io/projected/568b301b-486f-4cf6-8d4a-26598ba2fef0-kube-api-access-tll6v\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.841032 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.840971    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-maas-api-tls\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.841153 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.841072    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-metrics-certs\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.841209 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:06:06.841167    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-serving-cert: secret "maas-api-e2e-mt-314bd6fa-serving-cert" not found
Oct 30 19:06:06.841261 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:06:06.841239    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-maas-api-tls podName:568b301b-486f-4cf6-8d4a-26598ba2fef0 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:07.341217581 +0000 UTC m=+1762.451950846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-maas-api-tls") pod "maas-api-e2e-mt-314bd6fa-65c588785b-wdht7" (UID: "568b301b-486f-4cf6-8d4a-26598ba2fef0") : secret "maas-api-e2e-mt-314bd6fa-serving-cert" not found
Oct 30 19:06:06.841478 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.841456    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/568b301b-486f-4cf6-8d4a-26598ba2fef0-openshift-service-ca\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.843789 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.843764    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-metrics-certs\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:06.930500 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:06.930464    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tll6v\" (UniqueName: \"kubernetes.io/projected/568b301b-486f-4cf6-8d4a-26598ba2fef0-kube-api-access-tll6v\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:07.365238 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:07.365197    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-maas-api-tls\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:07.367971 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:07.367943    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-maas-api-tls\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-wdht7\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:07.489546 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:07.489508    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:07.865224 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:07.865188    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"]
Oct 30 19:06:07.865987 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:06:07.865962    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568b301b_486f_4cf6_8d4a_26598ba2fef0.slice/crio-f4d508a7a3caf23c0ee2cc1fac5b101cc0ef98b85426211116e928995b709495 WatchSource:0}: Error finding container f4d508a7a3caf23c0ee2cc1fac5b101cc0ef98b85426211116e928995b709495: Status 404 returned error can't find the container with id f4d508a7a3caf23c0ee2cc1fac5b101cc0ef98b85426211116e928995b709495
Oct 30 19:06:07.867655 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:07.867633    4391 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:06:08.875801 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:08.875733    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7" event={"ID":"568b301b-486f-4cf6-8d4a-26598ba2fef0","Type":"ContainerStarted","Data":"f4d508a7a3caf23c0ee2cc1fac5b101cc0ef98b85426211116e928995b709495"}
Oct 30 19:06:09.887003 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:09.886959    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7" event={"ID":"568b301b-486f-4cf6-8d4a-26598ba2fef0","Type":"ContainerStarted","Data":"60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424"}
Oct 30 19:06:09.887441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:09.887043    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:10.000283 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:10.000228    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7" podStartSLOduration=2.994558541 podStartE2EDuration="4.000213471s" totalImagesPullingTime="1.00565493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:09.997713325 +0000 UTC m=+1765.108446611" watchObservedRunningTime="2026-10-30 19:06:10.000213471 +0000 UTC m=+1765.110946755"
Oct 30 19:06:11.307090 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:06:11.307054    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:06:11.620765 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:11.620680    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"]
Oct 30 19:06:11.725141 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:11.725103    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"]
Oct 30 19:06:15.925475 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:15.925441    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:06:16.621426 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.621388    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw" podUID="93e57ff5-bec2-4f3c-9ff8-fedfad75ed33" containerName="payload-processing" containerID="cri-o://9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e" gracePeriod=25
Oct 30 19:06:16.725451 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.725369    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr" podUID="e4f95989-1ca6-443a-bf64-53eb4654f6b4" containerName="payload-pre-processing" containerID="cri-o://c875bf563b022b5fe416318c7b99c587ed7081f44c1f1f78509c0bae6974ad6c" gracePeriod=25
Oct 30 19:06:16.776913 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:06:16.776877    4391 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f95989_1ca6_443a_bf64_53eb4654f6b4.slice/crio-c875bf563b022b5fe416318c7b99c587ed7081f44c1f1f78509c0bae6974ad6c.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:06:16.913790 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.913762    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:06:16.966640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.966601    4391 generic.go:356] "Generic (PLEG): container finished" podID="e4f95989-1ca6-443a-bf64-53eb4654f6b4" containerID="c875bf563b022b5fe416318c7b99c587ed7081f44c1f1f78509c0bae6974ad6c" exitCode=0
Oct 30 19:06:16.967088 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.966610    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr" event={"ID":"e4f95989-1ca6-443a-bf64-53eb4654f6b4","Type":"ContainerDied","Data":"c875bf563b022b5fe416318c7b99c587ed7081f44c1f1f78509c0bae6974ad6c"}
Oct 30 19:06:16.968356 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.968329    4391 generic.go:356] "Generic (PLEG): container finished" podID="93e57ff5-bec2-4f3c-9ff8-fedfad75ed33" containerID="9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e" exitCode=0
Oct 30 19:06:16.968492 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.968377    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"
Oct 30 19:06:16.968492 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.968410    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw" event={"ID":"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33","Type":"ContainerDied","Data":"9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e"}
Oct 30 19:06:16.968492 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.968458    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw" event={"ID":"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33","Type":"ContainerDied","Data":"894e2b03fc5cfada31a6315efa8b89435bb652c62c67de8c43ab087968b251b5"}
Oct 30 19:06:16.968492 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.968482    4391 scope.go:122] "RemoveContainer" containerID="9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e"
Oct 30 19:06:16.982153 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.982128    4391 scope.go:122] "RemoveContainer" containerID="9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e"
Oct 30 19:06:16.982497 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:06:16.982470    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e\": container with ID starting with 9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e not found: ID does not exist" containerID="9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e"
Oct 30 19:06:16.982680 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:16.982509    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e"} err="failed to get container status \"9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e\": rpc error: code = NotFound desc = could not find container \"9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e\": container with ID starting with 9324e05951a526a6f4afc1f291322d4f70a45104594728aa2d137918b917541e not found: ID does not exist"
Oct 30 19:06:17.091613 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.072346    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-plugins-config-volume\") pod \"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33\" (UID: \"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33\") "
Oct 30 19:06:17.103735 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.103700    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-kube-api-access-dfqkq\" (UniqueName: \"kubernetes.io/projected/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-kube-api-access-dfqkq\") pod \"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33\" (UID: \"93e57ff5-bec2-4f3c-9ff8-fedfad75ed33\") "
Oct 30 19:06:17.150956 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.075029    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-plugins-config-volume" pod "93e57ff5-bec2-4f3c-9ff8-fedfad75ed33" (UID: "93e57ff5-bec2-4f3c-9ff8-fedfad75ed33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:17.150956 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.140640    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-kube-api-access-dfqkq" pod "93e57ff5-bec2-4f3c-9ff8-fedfad75ed33" (UID: "93e57ff5-bec2-4f3c-9ff8-fedfad75ed33"). InnerVolumeSpecName "kube-api-access-dfqkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:17.150956 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.145002    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:06:17.206510 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.206477    4391 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-plugins-config-volume\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:17.206510 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.206506    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqkq\" (UniqueName: \"kubernetes.io/projected/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33-kube-api-access-dfqkq\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:17.307172 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.307126    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f95989-1ca6-443a-bf64-53eb4654f6b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f95989-1ca6-443a-bf64-53eb4654f6b4-plugins-config-volume\") pod \"e4f95989-1ca6-443a-bf64-53eb4654f6b4\" (UID: \"e4f95989-1ca6-443a-bf64-53eb4654f6b4\") "
Oct 30 19:06:17.307369 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.307231    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f95989-1ca6-443a-bf64-53eb4654f6b4-kube-api-access-nffcw\" (UniqueName: \"kubernetes.io/projected/e4f95989-1ca6-443a-bf64-53eb4654f6b4-kube-api-access-nffcw\") pod \"e4f95989-1ca6-443a-bf64-53eb4654f6b4\" (UID: \"e4f95989-1ca6-443a-bf64-53eb4654f6b4\") "
Oct 30 19:06:17.307667 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.307629    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f95989-1ca6-443a-bf64-53eb4654f6b4-plugins-config-volume" pod "e4f95989-1ca6-443a-bf64-53eb4654f6b4" (UID: "e4f95989-1ca6-443a-bf64-53eb4654f6b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:17.308215 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.308187    4391 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f95989-1ca6-443a-bf64-53eb4654f6b4-plugins-config-volume\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:17.309537 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.309511    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f95989-1ca6-443a-bf64-53eb4654f6b4-kube-api-access-nffcw" pod "e4f95989-1ca6-443a-bf64-53eb4654f6b4" (UID: "e4f95989-1ca6-443a-bf64-53eb4654f6b4"). InnerVolumeSpecName "kube-api-access-nffcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:17.394677 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.394643    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"]
Oct 30 19:06:17.398988 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.398960    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4cef683f-6c67b56b-rhjjw"]
Oct 30 19:06:17.409343 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.409319    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nffcw\" (UniqueName: \"kubernetes.io/projected/e4f95989-1ca6-443a-bf64-53eb4654f6b4-kube-api-access-nffcw\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:17.978534 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.978502    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr" event={"ID":"e4f95989-1ca6-443a-bf64-53eb4654f6b4","Type":"ContainerDied","Data":"7c62c18ebabde3cdfdd5adf45e8857f9fc43a63911028481e2a0c4260a8048aa"}
Oct 30 19:06:17.978534 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.978518    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"
Oct 30 19:06:17.979073 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:17.978561    4391 scope.go:122] "RemoveContainer" containerID="c875bf563b022b5fe416318c7b99c587ed7081f44c1f1f78509c0bae6974ad6c"
Oct 30 19:06:18.142012 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:18.141975    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"]
Oct 30 19:06:18.188794 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:18.188761    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4cef683f-8d7b4dd67-cxnqr"]
Oct 30 19:06:19.008212 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:19.008178    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e57ff5-bec2-4f3c-9ff8-fedfad75ed33" path="/var/lib/kubelet/pods/93e57ff5-bec2-4f3c-9ff8-fedfad75ed33/volumes"
Oct 30 19:06:19.008694 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:19.008679    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f95989-1ca6-443a-bf64-53eb4654f6b4" path="/var/lib/kubelet/pods/e4f95989-1ca6-443a-bf64-53eb4654f6b4/volumes"
Oct 30 19:06:21.391805 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.391770    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"]
Oct 30 19:06:21.392416 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.392402    4391 cpu_manager.go:414] "containerMap: removing container" podUID="93e57ff5-bec2-4f3c-9ff8-fedfad75ed33" containerName="payload-processing"
Oct 30 19:06:21.392461 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.392417    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e57ff5-bec2-4f3c-9ff8-fedfad75ed33" containerName="payload-processing"
Oct 30 19:06:21.392461 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.392443    4391 cpu_manager.go:414] "containerMap: removing container" podUID="e4f95989-1ca6-443a-bf64-53eb4654f6b4" containerName="payload-pre-processing"
Oct 30 19:06:21.392461 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.392448    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f95989-1ca6-443a-bf64-53eb4654f6b4" containerName="payload-pre-processing"
Oct 30 19:06:21.392573 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.392563    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e57ff5-bec2-4f3c-9ff8-fedfad75ed33" containerName="payload-processing"
Oct 30 19:06:21.392632 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.392592    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f95989-1ca6-443a-bf64-53eb4654f6b4" containerName="payload-pre-processing"
Oct 30 19:06:21.396970 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.396949    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.469846 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.469812    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.470025 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.469888    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.470025 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.470006    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.470120 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.470067    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcm8\" (UniqueName: \"kubernetes.io/projected/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-kube-api-access-tpcm8\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.470165 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.470143    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.470235 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.470218    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.470271 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.470258    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.470307 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.470281    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.470394 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.470382    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.501055 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.501016    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"]
Oct 30 19:06:21.571808 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.571774    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.571996 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.571841    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.571996 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.571923    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.572114 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.571995    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcm8\" (UniqueName: \"kubernetes.io/projected/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-kube-api-access-tpcm8\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.572114 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.572045    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.572114 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.572093    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.572279 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.572138    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.572279 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.572175    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.572279 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.572220    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.572279 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.572235    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.572495 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.572430    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.572784 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.572631    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.572784 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.572706    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.573116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.573096    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.574725 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.574705    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.575106 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.575085    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.626832 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.626794    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.627002 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.626856    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcm8\" (UniqueName: \"kubernetes.io/projected/dc4a0ecf-d63c-4917-95e5-44998c76ef0a-kube-api-access-tpcm8\") pod \"maas-default-gateway-openshift-default-75d7c785f-59j4p\" (UID: \"dc4a0ecf-d63c-4917-95e5-44998c76ef0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:21.711703 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:21.711670    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:22.194362 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:22.194327    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"]
Oct 30 19:06:22.195500 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:06:22.195477    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4a0ecf_d63c_4917_95e5_44998c76ef0a.slice/crio-3551201e06aaca29d1a6d5ca159d85bef1afc76a96dc8fe5ad11060aace084bc WatchSource:0}: Error finding container 3551201e06aaca29d1a6d5ca159d85bef1afc76a96dc8fe5ad11060aace084bc: Status 404 returned error can't find the container with id 3551201e06aaca29d1a6d5ca159d85bef1afc76a96dc8fe5ad11060aace084bc
Oct 30 19:06:22.198932 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:22.198893    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:22.199027 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:22.198967    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:22.199027 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:22.198999    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:23.044194 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:23.044153    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" event={"ID":"dc4a0ecf-d63c-4917-95e5-44998c76ef0a","Type":"ContainerStarted","Data":"db86af97bb1ce4d92c816f53a6988dd676c9780eb1e4f864b0a55a1498cb101c"}
Oct 30 19:06:23.044645 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:23.044220    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" event={"ID":"dc4a0ecf-d63c-4917-95e5-44998c76ef0a","Type":"ContainerStarted","Data":"3551201e06aaca29d1a6d5ca159d85bef1afc76a96dc8fe5ad11060aace084bc"}
Oct 30 19:06:23.265168 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:23.265092    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" podStartSLOduration=2.265071336 podStartE2EDuration="2.265071336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:23.201427776 +0000 UTC m=+1778.312161059" watchObservedRunningTime="2026-10-30 19:06:23.265071336 +0000 UTC m=+1778.375804620"
Oct 30 19:06:23.712717 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:23.712680    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:23.714463 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:23.714441    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-59j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:06:23.714540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:23.714488    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" podUID="dc4a0ecf-d63c-4917-95e5-44998c76ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 19:06:24.713051 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:24.713012    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-59j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:06:24.713464 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:24.713070    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" podUID="dc4a0ecf-d63c-4917-95e5-44998c76ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 19:06:25.712831 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:25.712790    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-59j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:06:25.713016 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:25.712852    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" podUID="dc4a0ecf-d63c-4917-95e5-44998c76ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 19:06:25.875950 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:25.875915    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"]
Oct 30 19:06:25.876363 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:25.876132    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb" podUID="665346bc-6b95-4ba3-9b94-34c4cd020b1b" containerName="maas-api" containerID="cri-o://220090984a6568149a9ba3f8f4fd2323d08bb4b21adc754d887d85c822729046" gracePeriod=30
Oct 30 19:06:26.682277 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.682234    4391 generic.go:356] "Generic (PLEG): container finished" podID="665346bc-6b95-4ba3-9b94-34c4cd020b1b" containerID="220090984a6568149a9ba3f8f4fd2323d08bb4b21adc754d887d85c822729046" exitCode=0
Oct 30 19:06:26.682562 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.682543    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb" event={"ID":"665346bc-6b95-4ba3-9b94-34c4cd020b1b","Type":"ContainerDied","Data":"220090984a6568149a9ba3f8f4fd2323d08bb4b21adc754d887d85c822729046"}
Oct 30 19:06:26.711776 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.711743    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:06:26.712752 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.712715    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-59j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:06:26.712891 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.712769    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" podUID="dc4a0ecf-d63c-4917-95e5-44998c76ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 19:06:26.781400 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.781359    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-maas-api-tls\") pod \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") "
Oct 30 19:06:26.781613 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.781565    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665346bc-6b95-4ba3-9b94-34c4cd020b1b-kube-api-access-xpz5z\" (UniqueName: \"kubernetes.io/projected/665346bc-6b95-4ba3-9b94-34c4cd020b1b-kube-api-access-xpz5z\") pod \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") "
Oct 30 19:06:26.781669 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.781638    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/665346bc-6b95-4ba3-9b94-34c4cd020b1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665346bc-6b95-4ba3-9b94-34c4cd020b1b-openshift-service-ca\") pod \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") "
Oct 30 19:06:26.781739 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.781724    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-metrics-certs\") pod \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\" (UID: \"665346bc-6b95-4ba3-9b94-34c4cd020b1b\") "
Oct 30 19:06:26.782035 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.782007    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665346bc-6b95-4ba3-9b94-34c4cd020b1b-openshift-service-ca" pod "665346bc-6b95-4ba3-9b94-34c4cd020b1b" (UID: "665346bc-6b95-4ba3-9b94-34c4cd020b1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:26.782390 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.782365    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665346bc-6b95-4ba3-9b94-34c4cd020b1b-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:26.784127 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.784077    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665346bc-6b95-4ba3-9b94-34c4cd020b1b-kube-api-access-xpz5z" pod "665346bc-6b95-4ba3-9b94-34c4cd020b1b" (UID: "665346bc-6b95-4ba3-9b94-34c4cd020b1b"). InnerVolumeSpecName "kube-api-access-xpz5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:26.784127 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.784102    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-maas-api-tls" pod "665346bc-6b95-4ba3-9b94-34c4cd020b1b" (UID: "665346bc-6b95-4ba3-9b94-34c4cd020b1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:26.784238 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.784170    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-metrics-certs" pod "665346bc-6b95-4ba3-9b94-34c4cd020b1b" (UID: "665346bc-6b95-4ba3-9b94-34c4cd020b1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:26.900619 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.898947    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpz5z\" (UniqueName: \"kubernetes.io/projected/665346bc-6b95-4ba3-9b94-34c4cd020b1b-kube-api-access-xpz5z\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:26.900619 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.898991    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:26.900619 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:26.899004    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665346bc-6b95-4ba3-9b94-34c4cd020b1b-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:27.702679 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:27.702635    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb" event={"ID":"665346bc-6b95-4ba3-9b94-34c4cd020b1b","Type":"ContainerDied","Data":"a24723e312b3ad5c54d040b22e671d435576de94bd5d4c5e7ea7475ac4c3c4cc"}
Oct 30 19:06:27.702873 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:27.702729    4391 scope.go:122] "RemoveContainer" containerID="220090984a6568149a9ba3f8f4fd2323d08bb4b21adc754d887d85c822729046"
Oct 30 19:06:27.702873 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:27.702654    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"
Oct 30 19:06:27.712932 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:27.712895    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-59j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:06:27.713066 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:27.712957    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" podUID="dc4a0ecf-d63c-4917-95e5-44998c76ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 19:06:27.878011 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:27.877975    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"]
Oct 30 19:06:27.882053 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:27.882021    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06203ab6-5cb87688cc-gf2tb"]
Oct 30 19:06:28.312099 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:28.312062    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:06:28.712041 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:28.712001    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-59j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:06:28.712245 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:28.712063    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" podUID="dc4a0ecf-d63c-4917-95e5-44998c76ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 19:06:29.007520 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:29.007440    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665346bc-6b95-4ba3-9b94-34c4cd020b1b" path="/var/lib/kubelet/pods/665346bc-6b95-4ba3-9b94-34c4cd020b1b/volumes"
Oct 30 19:06:29.712549 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:29.712507    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-59j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:06:29.713109 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:29.712610    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" podUID="dc4a0ecf-d63c-4917-95e5-44998c76ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 19:06:30.712633 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:30.712596    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-59j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:06:30.713014 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:30.712665    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p" podUID="dc4a0ecf-d63c-4917-95e5-44998c76ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 19:06:32.147041 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:32.147008    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:32.147540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:32.147288    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:32.148373 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:32.148353    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-59j4p"
Oct 30 19:06:41.255063 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:06:41.255022    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:06:45.510094 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:45.510048    4391 scope.go:122] "RemoveContainer" containerID="0861c24ae730fb5f17d04328b7e39f89c1227b92212808d87f545a0732d59376"
Oct 30 19:06:45.520550 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:45.520527    4391 scope.go:122] "RemoveContainer" containerID="bc719d32beffc5a3ef79a8c47bff4482693c8a4bd94de9315db6c1f1dbeed3fe"
Oct 30 19:06:45.531277 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:45.531253    4391 scope.go:122] "RemoveContainer" containerID="46ff756e4cfe2b06d2136c2d84c1e25251dc5028874545699d45c3f0673c375f"
Oct 30 19:06:51.648818 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.648780    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"]
Oct 30 19:06:51.649710 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.649686    4391 cpu_manager.go:414] "containerMap: removing container" podUID="665346bc-6b95-4ba3-9b94-34c4cd020b1b" containerName="maas-api"
Oct 30 19:06:51.649710 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.649709    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665346bc-6b95-4ba3-9b94-34c4cd020b1b" containerName="maas-api"
Oct 30 19:06:51.649850 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.649837    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="665346bc-6b95-4ba3-9b94-34c4cd020b1b" containerName="maas-api"
Oct 30 19:06:51.655455 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.655433    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.675307 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.675268    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.675473 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.675415    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.675535 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.675481    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qttmr\" (UniqueName: \"kubernetes.io/projected/d4909aee-0ac5-4527-a9d9-976006e9a1cc-kube-api-access-qttmr\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.675617 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.675598    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.675672 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.675657    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.675775 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.675758    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.675824 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.675802    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.675883 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.675869    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.675930 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.675905    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.777291 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.777227    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.777523 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.777332    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.777523 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.777389    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qttmr\" (UniqueName: \"kubernetes.io/projected/d4909aee-0ac5-4527-a9d9-976006e9a1cc-kube-api-access-qttmr\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.777523 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.777430    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.777523 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.777487    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.777809 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.777559    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.777809 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.777621    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.777809 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.777650    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.777809 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.777689    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.778118 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.778093    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.778428 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.778398    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.778798 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.778412    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.778920 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.778701    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.778986 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.778911    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.780907 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.780882    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.781002 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.780905    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.833851 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.833822    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qttmr\" (UniqueName: \"kubernetes.io/projected/d4909aee-0ac5-4527-a9d9-976006e9a1cc-kube-api-access-qttmr\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.834000 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.833824    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4909aee-0ac5-4527-a9d9-976006e9a1cc-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t8j9r\" (UID: \"d4909aee-0ac5-4527-a9d9-976006e9a1cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:51.851380 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.851344    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"]
Oct 30 19:06:51.970201 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:51.970162    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:52.394438 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:52.394346    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"]
Oct 30 19:06:52.428222 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:06:52.428191    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4909aee_0ac5_4527_a9d9_976006e9a1cc.slice/crio-485363d15a8b04a9429cb50ed4ce07de02d1447d7807e46e0d1191fcec6d5ba7 WatchSource:0}: Error finding container 485363d15a8b04a9429cb50ed4ce07de02d1447d7807e46e0d1191fcec6d5ba7: Status 404 returned error can't find the container with id 485363d15a8b04a9429cb50ed4ce07de02d1447d7807e46e0d1191fcec6d5ba7
Oct 30 19:06:52.430850 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:52.430814    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:52.430959 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:52.430905    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:52.430959 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:52.430952    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:53.009715 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:53.009624    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r" event={"ID":"d4909aee-0ac5-4527-a9d9-976006e9a1cc","Type":"ContainerStarted","Data":"d3e751e951036c1d5c9cceefb3b13248ea3cdec5d4a184b174e47173b8700600"}
Oct 30 19:06:53.309028 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:53.262008    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r" event={"ID":"d4909aee-0ac5-4527-a9d9-976006e9a1cc","Type":"ContainerStarted","Data":"485363d15a8b04a9429cb50ed4ce07de02d1447d7807e46e0d1191fcec6d5ba7"}
Oct 30 19:06:53.415360 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:53.415293    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r" podStartSLOduration=2.415271322 podStartE2EDuration="2.415271322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:53.409210698 +0000 UTC m=+1808.519943981" watchObservedRunningTime="2026-10-30 19:06:53.415271322 +0000 UTC m=+1808.526004606"
Oct 30 19:06:53.974268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:53.973030    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:06:53.974268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:53.974193    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t8j9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:06:53.974268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:53.974247    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r" podUID="d4909aee-0ac5-4527-a9d9-976006e9a1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:06:54.980607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:54.979678    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t8j9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:06:54.980607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:54.979741    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r" podUID="d4909aee-0ac5-4527-a9d9-976006e9a1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:06:55.971899 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:55.971857    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t8j9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:06:55.972102 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:55.971924    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r" podUID="d4909aee-0ac5-4527-a9d9-976006e9a1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:06:56.971825 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:56.971786    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t8j9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:06:56.972344 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:56.971861    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r" podUID="d4909aee-0ac5-4527-a9d9-976006e9a1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:06:57.972034 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:57.971991    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t8j9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:06:57.972646 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:57.972074    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r" podUID="d4909aee-0ac5-4527-a9d9-976006e9a1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:06:58.972065 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:58.972027    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t8j9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:06:58.972515 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:58.972090    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r" podUID="d4909aee-0ac5-4527-a9d9-976006e9a1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:06:59.627480 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.627438    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"]
Oct 30 19:06:59.633481 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.633442    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.685069 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.685037    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"]
Oct 30 19:06:59.725033 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.725002    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-0f3f2f-serving-cert\""
Oct 30 19:06:59.785843 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.785491    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.786760 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.786727    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5pxl\" (UniqueName: \"kubernetes.io/projected/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-kube-api-access-t5pxl\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.787120 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.787097    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.787295 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.787275    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.901659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.898651    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.901659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.898741    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.901659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.898918    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.901659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.898984    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pxl\" (UniqueName: \"kubernetes.io/projected/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-kube-api-access-t5pxl\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.901659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.901178    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.901659 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:06:59.901300    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-serving-cert: secret "maas-api-e2e-shared-b-w6-0f3f2f-serving-cert" not found
Oct 30 19:06:59.901659 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:06:59.901365    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-maas-api-tls podName:4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1 nodeName:}" failed. No retries permitted until 2026-10-30 19:07:00.401343855 +0000 UTC m=+1815.512077122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-maas-api-tls") pod "maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf" (UID: "4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1") : secret "maas-api-e2e-shared-b-w6-0f3f2f-serving-cert" not found
Oct 30 19:06:59.930817 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.930785    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.990433 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.986426    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pxl\" (UniqueName: \"kubernetes.io/projected/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-kube-api-access-t5pxl\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:06:59.990433 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.987882    4391 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t8j9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:06:59.990433 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:06:59.987960    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r" podUID="d4909aee-0ac5-4527-a9d9-976006e9a1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 19:07:00.408174 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:00.408137    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:07:00.410629 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:00.410602    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:07:00.550112 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:00.550078    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:07:00.932813 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:00.932779    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"]
Oct 30 19:07:01.216423 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:01.216317    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf" event={"ID":"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1","Type":"ContainerStarted","Data":"d6713fbc79fe61af4279eff0aca759de9fe98b7b200a1fab5c6f1739f91bda40"}
Oct 30 19:07:01.754715 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:01.754681    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:07:01.755241 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:01.755212    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:07:01.756291 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:01.756268    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t8j9r"
Oct 30 19:07:02.226205 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:02.226169    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf" event={"ID":"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1","Type":"ContainerStarted","Data":"c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d"}
Oct 30 19:07:02.226705 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:02.226390    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:07:02.335848 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:02.335783    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf" podStartSLOduration=2.4917886830000002 podStartE2EDuration="3.335768312s" totalImagesPullingTime="843.979629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:02.335739964 +0000 UTC m=+1817.446473246" watchObservedRunningTime="2026-10-30 19:07:02.335768312 +0000 UTC m=+1817.446501594"
Oct 30 19:07:08.238859 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:08.238825    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:07:09.787603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:09.787238    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:07:11.254644 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:07:11.254600    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:07:27.999658 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:27.999540    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"]
Oct 30 19:07:28.000227 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:27.999764    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy" containerID="cri-o://b1c56cffdce3543d5c55e19286e59b4b859940308ade9470b9c6ce9ecd121a87" gracePeriod=30
Oct 30 19:07:32.233960 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.233914    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"]
Oct 30 19:07:32.239640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.239619    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.297342 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.297303    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"]
Oct 30 19:07:32.334443 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.334392    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-dshm\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.334637 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.334534    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wslr6\" (UniqueName: \"kubernetes.io/projected/a1dc6b43-837b-4600-b34e-76cf341240bd-kube-api-access-wslr6\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.334637 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.334614    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-home\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.334816 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.334798    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-tmp-dir\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.334858 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.334841    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-model-cache\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.335073 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.335046    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1dc6b43-837b-4600-b34e-76cf341240bd-tls-certs\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.337965 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.337939    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-314bd6fa\"/\"e2e-lifecycle-model-314bd6fa-kserve-self-signed-certs\""
Oct 30 19:07:32.337965 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.337946    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-314bd6fa\"/\"openshift-service-ca.crt\""
Oct 30 19:07:32.338118 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.338037    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-314bd6fa\"/\"default-dockercfg-hzcdr\""
Oct 30 19:07:32.338174 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.338158    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-314bd6fa\"/\"kube-root-ca.crt\""
Oct 30 19:07:32.447637 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.438879    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1dc6b43-837b-4600-b34e-76cf341240bd-tls-certs\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.447637 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.439044    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-dshm\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.447637 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.439122    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wslr6\" (UniqueName: \"kubernetes.io/projected/a1dc6b43-837b-4600-b34e-76cf341240bd-kube-api-access-wslr6\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.447637 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.439175    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-home\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.447637 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.439292    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-tmp-dir\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.447637 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.439323    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-model-cache\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.447637 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.439884    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-model-cache\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.449353 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.448263    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1dc6b43-837b-4600-b34e-76cf341240bd-tls-certs\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.449353 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.448487    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-home\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.449353 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.448556    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-tmp-dir\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.470294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.468753    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-dshm\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.524485 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.522756    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wslr6\" (UniqueName: \"kubernetes.io/projected/a1dc6b43-837b-4600-b34e-76cf341240bd-kube-api-access-wslr6\") pod \"e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") " pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:32.552905 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:32.552868    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:33.092160 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:33.092128    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"]
Oct 30 19:07:33.092901 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:07:33.092871    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1dc6b43_837b_4600_b34e_76cf341240bd.slice/crio-2ffce981543990578d470308999e812e28bd78685c5387ae2e9a77b37fbbbe40 WatchSource:0}: Error finding container 2ffce981543990578d470308999e812e28bd78685c5387ae2e9a77b37fbbbe40: Status 404 returned error can't find the container with id 2ffce981543990578d470308999e812e28bd78685c5387ae2e9a77b37fbbbe40
Oct 30 19:07:33.924826 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:33.924788    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x" event={"ID":"a1dc6b43-837b-4600-b34e-76cf341240bd","Type":"ContainerStarted","Data":"de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184"}
Oct 30 19:07:33.925301 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:33.924841    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x" event={"ID":"a1dc6b43-837b-4600-b34e-76cf341240bd","Type":"ContainerStarted","Data":"2ffce981543990578d470308999e812e28bd78685c5387ae2e9a77b37fbbbe40"}
Oct 30 19:07:33.926837 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:33.926737    4391 generic.go:356] "Generic (PLEG): container finished" podID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerID="b1c56cffdce3543d5c55e19286e59b4b859940308ade9470b9c6ce9ecd121a87" exitCode=0
Oct 30 19:07:33.926837 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:33.926809    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" event={"ID":"aec861e9-b8ce-49e6-92b2-ba55d21354c4","Type":"ContainerDied","Data":"b1c56cffdce3543d5c55e19286e59b4b859940308ade9470b9c6ce9ecd121a87"}
Oct 30 19:07:33.958294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:33.958263    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 19:07:34.043161 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.043103    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x" podStartSLOduration=2.043085269 podStartE2EDuration="2.043085269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:34.038922667 +0000 UTC m=+1849.149655950" watchObservedRunningTime="2026-10-30 19:07:34.043085269 +0000 UTC m=+1849.153818553"
Oct 30 19:07:34.049596 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.049555    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-podinfo\") pod \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") "
Oct 30 19:07:34.049760 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.049612    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-data\") pod \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") "
Oct 30 19:07:34.049760 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.049704    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-socket\") pod \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") "
Oct 30 19:07:34.049760 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.049753    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-kube-api-access-sbq68\" (UniqueName: \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-kube-api-access-sbq68\") pod \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") "
Oct 30 19:07:34.049926 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.049845    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istiod-ca-cert\") pod \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") "
Oct 30 19:07:34.049926 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.049869    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-data" pod "aec861e9-b8ce-49e6-92b2-ba55d21354c4" (UID: "aec861e9-b8ce-49e6-92b2-ba55d21354c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:34.050034 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.049959    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-envoy\") pod \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") "
Oct 30 19:07:34.050034 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.049986    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-socket" pod "aec861e9-b8ce-49e6-92b2-ba55d21354c4" (UID: "aec861e9-b8ce-49e6-92b2-ba55d21354c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:34.050034 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.049996    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-certs\") pod \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") "
Oct 30 19:07:34.050180 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.050144    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-credential-socket\") pod \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") "
Oct 30 19:07:34.050230 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.050193    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-token\" (UniqueName: \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-token\") pod \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\" (UID: \"aec861e9-b8ce-49e6-92b2-ba55d21354c4\") "
Oct 30 19:07:34.050230 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.050197    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-certs" pod "aec861e9-b8ce-49e6-92b2-ba55d21354c4" (UID: "aec861e9-b8ce-49e6-92b2-ba55d21354c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:34.050230 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.050207    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istiod-ca-cert" pod "aec861e9-b8ce-49e6-92b2-ba55d21354c4" (UID: "aec861e9-b8ce-49e6-92b2-ba55d21354c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:34.050399 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.050375    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-credential-socket" pod "aec861e9-b8ce-49e6-92b2-ba55d21354c4" (UID: "aec861e9-b8ce-49e6-92b2-ba55d21354c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:34.051186 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.051155    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:34.051302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.051189    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:34.051302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.051205    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:34.051302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.051220    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:34.051302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.051235    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:34.052135 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.052109    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-podinfo" pod "aec861e9-b8ce-49e6-92b2-ba55d21354c4" (UID: "aec861e9-b8ce-49e6-92b2-ba55d21354c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:07:34.053077 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.053051    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-kube-api-access-sbq68" pod "aec861e9-b8ce-49e6-92b2-ba55d21354c4" (UID: "aec861e9-b8ce-49e6-92b2-ba55d21354c4"). InnerVolumeSpecName "kube-api-access-sbq68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:34.053213 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.053111    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-token" pod "aec861e9-b8ce-49e6-92b2-ba55d21354c4" (UID: "aec861e9-b8ce-49e6-92b2-ba55d21354c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:34.053213 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.053145    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-envoy" pod "aec861e9-b8ce-49e6-92b2-ba55d21354c4" (UID: "aec861e9-b8ce-49e6-92b2-ba55d21354c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:34.153232 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.153141    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:34.153466 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.153445    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:34.153602 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.153569    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbq68\" (UniqueName: \"kubernetes.io/projected/aec861e9-b8ce-49e6-92b2-ba55d21354c4-kube-api-access-sbq68\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:34.153709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.153686    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec861e9-b8ce-49e6-92b2-ba55d21354c4-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:34.554213 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.554176    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:34.978604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.975950    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4" event={"ID":"aec861e9-b8ce-49e6-92b2-ba55d21354c4","Type":"ContainerDied","Data":"c06a354c7d11f541db56ebb5935ec31673e1d5e17abc6516b85ca3539493c68b"}
Oct 30 19:07:34.978604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.976024    4391 scope.go:122] "RemoveContainer" containerID="b1c56cffdce3543d5c55e19286e59b4b859940308ade9470b9c6ce9ecd121a87"
Oct 30 19:07:34.978604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:34.976220    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"
Oct 30 19:07:35.281838 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:35.281753    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"]
Oct 30 19:07:35.285798 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:35.285769    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-2r7z4"]
Oct 30 19:07:37.007213 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:37.007178    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" path="/var/lib/kubelet/pods/aec861e9-b8ce-49e6-92b2-ba55d21354c4/volumes"
Oct 30 19:07:41.253341 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:07:41.253298    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:07:42.553912 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:42.553878    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:42.566479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:42.566453    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:43.048488 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:43.048460    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:07:56.390899 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:07:56.390848    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:08:00.343272 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.343230    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"]
Oct 30 19:08:00.344122 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.344101    4391 cpu_manager.go:414] "containerMap: removing container" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy"
Oct 30 19:08:00.344180 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.344124    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy"
Oct 30 19:08:00.344267 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.344258    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec861e9-b8ce-49e6-92b2-ba55d21354c4" containerName="istio-proxy"
Oct 30 19:08:00.349268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.349244    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.417620 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.417573    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"]
Oct 30 19:08:00.456344 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.455833    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-05565d-serving-cert\""
Oct 30 19:08:00.522037 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.522003    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-maas-api-tls\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.522530 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.522499    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71279088-1baf-4ac7-a020-a22ce6f5f76b-openshift-service-ca\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.523155 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.523134    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-metrics-certs\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.523448 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.523407    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txspj\" (UniqueName: \"kubernetes.io/projected/71279088-1baf-4ac7-a020-a22ce6f5f76b-kube-api-access-txspj\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.646287 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.644799    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-metrics-certs\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.646287 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.644914    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txspj\" (UniqueName: \"kubernetes.io/projected/71279088-1baf-4ac7-a020-a22ce6f5f76b-kube-api-access-txspj\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.646287 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.644983    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-maas-api-tls\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.646287 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.645075    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71279088-1baf-4ac7-a020-a22ce6f5f76b-openshift-service-ca\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.646697 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.646297    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71279088-1baf-4ac7-a020-a22ce6f5f76b-openshift-service-ca\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.664606 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:00.658766    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-serving-cert: secret "maas-api-e2e-api-a-05565d-serving-cert" not found
Oct 30 19:08:00.664606 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:00.658876    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-maas-api-tls podName:71279088-1baf-4ac7-a020-a22ce6f5f76b nodeName:}" failed. No retries permitted until 2026-10-30 19:08:01.158852529 +0000 UTC m=+1876.269585803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-maas-api-tls") pod "maas-api-e2e-api-a-05565d-bf88f6449-sq8x8" (UID: "71279088-1baf-4ac7-a020-a22ce6f5f76b") : secret "maas-api-e2e-api-a-05565d-serving-cert" not found
Oct 30 19:08:00.699660 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.689076    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-metrics-certs\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:00.774613 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:00.774563    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txspj\" (UniqueName: \"kubernetes.io/projected/71279088-1baf-4ac7-a020-a22ce6f5f76b-kube-api-access-txspj\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:01.165604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:01.164870    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-maas-api-tls\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:01.195605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:01.195509    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-maas-api-tls\") pod \"maas-api-e2e-api-a-05565d-bf88f6449-sq8x8\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:01.286602 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:01.286076    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:01.880862 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:01.871771    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"]
Oct 30 19:08:02.325912 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:02.325799    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8" event={"ID":"71279088-1baf-4ac7-a020-a22ce6f5f76b","Type":"ContainerStarted","Data":"84e15590da5d34d67c041ee295563d8236ddf65750e4f79b1acec174c5dbc480"}
Oct 30 19:08:03.337697 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:03.337650    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8" event={"ID":"71279088-1baf-4ac7-a020-a22ce6f5f76b","Type":"ContainerStarted","Data":"8726739591f36f4551c869beb7ff04d4af8d39e4d1a9193db97c9eac2601353c"}
Oct 30 19:08:03.338179 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:03.337769    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:03.460172 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:03.460105    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8" podStartSLOduration=2.5637688389999997 podStartE2EDuration="3.460081783s" totalImagesPullingTime="896.312944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:03.455985493 +0000 UTC m=+1878.566718775" watchObservedRunningTime="2026-10-30 19:08:03.460081783 +0000 UTC m=+1878.570815068"
Oct 30 19:08:09.350283 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:09.350248    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:08:11.253531 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:11.253494    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:08:23.863098 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:23.863061    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8876-2d1e2c"]
Oct 30 19:08:23.867658 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:23.867639    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-2d1e2c"
Oct 30 19:08:23.926425 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:23.926396    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qk8z9\""
Oct 30 19:08:23.928087 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:23.928065    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmw9h\" (UniqueName: \"kubernetes.io/projected/982d0dd6-976f-47d9-98f2-9316f17f4bce-kube-api-access-tmw9h\") pod \"test-curl-8876-2d1e2c\" (UID: \"982d0dd6-976f-47d9-98f2-9316f17f4bce\") " pod="opendatahub/test-curl-8876-2d1e2c"
Oct 30 19:08:23.973116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:23.973081    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-2d1e2c"]
Oct 30 19:08:24.029918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:24.029880    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmw9h\" (UniqueName: \"kubernetes.io/projected/982d0dd6-976f-47d9-98f2-9316f17f4bce-kube-api-access-tmw9h\") pod \"test-curl-8876-2d1e2c\" (UID: \"982d0dd6-976f-47d9-98f2-9316f17f4bce\") " pod="opendatahub/test-curl-8876-2d1e2c"
Oct 30 19:08:24.085983 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:24.085944    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmw9h\" (UniqueName: \"kubernetes.io/projected/982d0dd6-976f-47d9-98f2-9316f17f4bce-kube-api-access-tmw9h\") pod \"test-curl-8876-2d1e2c\" (UID: \"982d0dd6-976f-47d9-98f2-9316f17f4bce\") " pod="opendatahub/test-curl-8876-2d1e2c"
Oct 30 19:08:24.180325 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:24.180286    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-2d1e2c"
Oct 30 19:08:24.567870 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:24.567831    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-2d1e2c"]
Oct 30 19:08:24.568192 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:08:24.568164    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod982d0dd6_976f_47d9_98f2_9316f17f4bce.slice/crio-20d4863439d6f73a1e79f3ace9d8c038ddae90231df7f9461eb90f394ee832a9 WatchSource:0}: Error finding container 20d4863439d6f73a1e79f3ace9d8c038ddae90231df7f9461eb90f394ee832a9: Status 404 returned error can't find the container with id 20d4863439d6f73a1e79f3ace9d8c038ddae90231df7f9461eb90f394ee832a9
Oct 30 19:08:25.500246 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:25.500205    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-2d1e2c" event={"ID":"982d0dd6-976f-47d9-98f2-9316f17f4bce","Type":"ContainerStarted","Data":"12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f"}
Oct 30 19:08:25.500699 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:25.500257    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-2d1e2c" event={"ID":"982d0dd6-976f-47d9-98f2-9316f17f4bce","Type":"ContainerStarted","Data":"20d4863439d6f73a1e79f3ace9d8c038ddae90231df7f9461eb90f394ee832a9"}
Oct 30 19:08:25.616543 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:25.616491    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8876-2d1e2c" podStartSLOduration=2.616477064 podStartE2EDuration="2.616477064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:25.615794995 +0000 UTC m=+1900.726528278" watchObservedRunningTime="2026-10-30 19:08:25.616477064 +0000 UTC m=+1900.727210347"
Oct 30 19:08:27.167814 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.167772    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8876-2d1e2c"]
Oct 30 19:08:27.172838 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.172804    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8876-2d1e2c"]
Oct 30 19:08:27.332162 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.332124    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8876-98a868"]
Oct 30 19:08:27.332831 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.332817    4391 cpu_manager.go:414] "containerMap: removing container" podUID="982d0dd6-976f-47d9-98f2-9316f17f4bce" containerName="test-curl-8876-2d1e2c"
Oct 30 19:08:27.332886 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.332833    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982d0dd6-976f-47d9-98f2-9316f17f4bce" containerName="test-curl-8876-2d1e2c"
Oct 30 19:08:27.332985 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.332975    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="982d0dd6-976f-47d9-98f2-9316f17f4bce" containerName="test-curl-8876-2d1e2c"
Oct 30 19:08:27.336771 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.336753    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-98a868"
Oct 30 19:08:27.387390 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.387349    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76ncp\" (UniqueName: \"kubernetes.io/projected/731d0671-acef-4e2f-8dd6-c7cca3ace772-kube-api-access-76ncp\") pod \"test-curl-8876-98a868\" (UID: \"731d0671-acef-4e2f-8dd6-c7cca3ace772\") " pod="opendatahub/test-curl-8876-98a868"
Oct 30 19:08:27.409620 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.409531    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-98a868"]
Oct 30 19:08:27.493879 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.493777    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ncp\" (UniqueName: \"kubernetes.io/projected/731d0671-acef-4e2f-8dd6-c7cca3ace772-kube-api-access-76ncp\") pod \"test-curl-8876-98a868\" (UID: \"731d0671-acef-4e2f-8dd6-c7cca3ace772\") " pod="opendatahub/test-curl-8876-98a868"
Oct 30 19:08:27.516037 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.515998    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8876-2d1e2c" podUID="982d0dd6-976f-47d9-98f2-9316f17f4bce" containerName="test-curl-8876-2d1e2c" containerID="cri-o://12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f" gracePeriod=2
Oct 30 19:08:27.548988 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.548957    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ncp\" (UniqueName: \"kubernetes.io/projected/731d0671-acef-4e2f-8dd6-c7cca3ace772-kube-api-access-76ncp\") pod \"test-curl-8876-98a868\" (UID: \"731d0671-acef-4e2f-8dd6-c7cca3ace772\") " pod="opendatahub/test-curl-8876-98a868"
Oct 30 19:08:27.564146 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:27.564106    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-2d1e2c\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="982d0dd6-976f-47d9-98f2-9316f17f4bce" pod="opendatahub/test-curl-8876-2d1e2c"
Oct 30 19:08:27.650539 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:27.650502    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-98a868"
Oct 30 19:08:28.037837 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:28.037801    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-98a868"]
Oct 30 19:08:28.038460 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:08:28.038434    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod731d0671_acef_4e2f_8dd6_c7cca3ace772.slice/crio-788f45c414d9dd406df91f9a975ef3a5df749c5b4f09471fce2790e099db4c13 WatchSource:0}: Error finding container 788f45c414d9dd406df91f9a975ef3a5df749c5b4f09471fce2790e099db4c13: Status 404 returned error can't find the container with id 788f45c414d9dd406df91f9a975ef3a5df749c5b4f09471fce2790e099db4c13
Oct 30 19:08:28.525519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:28.525483    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-98a868" event={"ID":"731d0671-acef-4e2f-8dd6-c7cca3ace772","Type":"ContainerStarted","Data":"0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4"}
Oct 30 19:08:28.525519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:28.525525    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-98a868" event={"ID":"731d0671-acef-4e2f-8dd6-c7cca3ace772","Type":"ContainerStarted","Data":"788f45c414d9dd406df91f9a975ef3a5df749c5b4f09471fce2790e099db4c13"}
Oct 30 19:08:28.639733 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:28.639660    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8876-98a868" podStartSLOduration=1.63964642 podStartE2EDuration="1.63964642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:28.634966824 +0000 UTC m=+1903.745700106" watchObservedRunningTime="2026-10-30 19:08:28.63964642 +0000 UTC m=+1903.750379703"
Oct 30 19:08:28.683154 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:28.683115    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-2d1e2c\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="982d0dd6-976f-47d9-98f2-9316f17f4bce" pod="opendatahub/test-curl-8876-2d1e2c"
Oct 30 19:08:29.864685 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:29.864651    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-2d1e2c"
Oct 30 19:08:29.969518 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:29.969483    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982d0dd6-976f-47d9-98f2-9316f17f4bce-kube-api-access-tmw9h\" (UniqueName: \"kubernetes.io/projected/982d0dd6-976f-47d9-98f2-9316f17f4bce-kube-api-access-tmw9h\") pod \"982d0dd6-976f-47d9-98f2-9316f17f4bce\" (UID: \"982d0dd6-976f-47d9-98f2-9316f17f4bce\") "
Oct 30 19:08:29.971674 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:29.971645    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982d0dd6-976f-47d9-98f2-9316f17f4bce-kube-api-access-tmw9h" pod "982d0dd6-976f-47d9-98f2-9316f17f4bce" (UID: "982d0dd6-976f-47d9-98f2-9316f17f4bce"). InnerVolumeSpecName "kube-api-access-tmw9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:30.071841 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.071807    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmw9h\" (UniqueName: \"kubernetes.io/projected/982d0dd6-976f-47d9-98f2-9316f17f4bce-kube-api-access-tmw9h\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:30.323602 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.321689    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8876-98a868"]
Oct 30 19:08:30.329534 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.329500    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8876-98a868"]
Oct 30 19:08:30.492513 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.492480    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8876-e851f1"]
Oct 30 19:08:30.493222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.493206    4391 cpu_manager.go:414] "containerMap: removing container" podUID="731d0671-acef-4e2f-8dd6-c7cca3ace772" containerName="test-curl-8876-98a868"
Oct 30 19:08:30.493270 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.493224    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731d0671-acef-4e2f-8dd6-c7cca3ace772" containerName="test-curl-8876-98a868"
Oct 30 19:08:30.493347 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.493335    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="731d0671-acef-4e2f-8dd6-c7cca3ace772" containerName="test-curl-8876-98a868"
Oct 30 19:08:30.496778 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.496758    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:30.542242 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.542209    4391 generic.go:356] "Generic (PLEG): container finished" podID="982d0dd6-976f-47d9-98f2-9316f17f4bce" containerID="12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f" exitCode=137
Oct 30 19:08:30.542358 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.542252    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-2d1e2c"
Oct 30 19:08:30.542358 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.542312    4391 scope.go:122] "RemoveContainer" containerID="12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f"
Oct 30 19:08:30.542534 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.542507    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8876-98a868" podUID="731d0671-acef-4e2f-8dd6-c7cca3ace772" containerName="test-curl-8876-98a868" containerID="cri-o://0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4" gracePeriod=2
Oct 30 19:08:30.553133 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.553014    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-e851f1"]
Oct 30 19:08:30.555303 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.555252    4391 scope.go:122] "RemoveContainer" containerID="12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f"
Oct 30 19:08:30.555647 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:30.555616    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f\": container with ID starting with 12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f not found: ID does not exist" containerID="12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f"
Oct 30 19:08:30.555740 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.555658    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f"} err="failed to get container status \"12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f\": rpc error: code = NotFound desc = could not find container \"12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f\": container with ID starting with 12f31a2aef6d6f851e4f1789c1304a04ac1c3a8dce601fcb4c933cab96e84f1f not found: ID does not exist"
Oct 30 19:08:30.587232 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.587143    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xhtc\" (UniqueName: \"kubernetes.io/projected/6375b445-3df4-4d1a-be47-6650607365c9-kube-api-access-4xhtc\") pod \"test-curl-8876-e851f1\" (UID: \"6375b445-3df4-4d1a-be47-6650607365c9\") " pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:30.664536 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:30.664496    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-98a868\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="731d0671-acef-4e2f-8dd6-c7cca3ace772" pod="opendatahub/test-curl-8876-98a868"
Oct 30 19:08:30.688338 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.688306    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xhtc\" (UniqueName: \"kubernetes.io/projected/6375b445-3df4-4d1a-be47-6650607365c9-kube-api-access-4xhtc\") pod \"test-curl-8876-e851f1\" (UID: \"6375b445-3df4-4d1a-be47-6650607365c9\") " pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:30.715625 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:30.715560    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-2d1e2c\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="982d0dd6-976f-47d9-98f2-9316f17f4bce" pod="opendatahub/test-curl-8876-2d1e2c"
Oct 30 19:08:30.745832 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.745803    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xhtc\" (UniqueName: \"kubernetes.io/projected/6375b445-3df4-4d1a-be47-6650607365c9-kube-api-access-4xhtc\") pod \"test-curl-8876-e851f1\" (UID: \"6375b445-3df4-4d1a-be47-6650607365c9\") " pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:30.809496 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:30.809465    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:31.008315 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:31.008283    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982d0dd6-976f-47d9-98f2-9316f17f4bce" path="/var/lib/kubelet/pods/982d0dd6-976f-47d9-98f2-9316f17f4bce/volumes"
Oct 30 19:08:31.299620 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:31.299571    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-e851f1"]
Oct 30 19:08:31.299778 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:08:31.299735    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6375b445_3df4_4d1a_be47_6650607365c9.slice/crio-8893ef496b1a5ee117e1fe2dcc63eeea50bbc930e2e034c6dcc022ce6ecc14b3 WatchSource:0}: Error finding container 8893ef496b1a5ee117e1fe2dcc63eeea50bbc930e2e034c6dcc022ce6ecc14b3: Status 404 returned error can't find the container with id 8893ef496b1a5ee117e1fe2dcc63eeea50bbc930e2e034c6dcc022ce6ecc14b3
Oct 30 19:08:31.551649 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:31.551531    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-e851f1" event={"ID":"6375b445-3df4-4d1a-be47-6650607365c9","Type":"ContainerStarted","Data":"a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049"}
Oct 30 19:08:31.551649 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:31.551595    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-e851f1" event={"ID":"6375b445-3df4-4d1a-be47-6650607365c9","Type":"ContainerStarted","Data":"8893ef496b1a5ee117e1fe2dcc63eeea50bbc930e2e034c6dcc022ce6ecc14b3"}
Oct 30 19:08:31.664998 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:31.664927    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8876-e851f1" podStartSLOduration=1.664909866 podStartE2EDuration="1.664909866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:31.66040965 +0000 UTC m=+1906.771142932" watchObservedRunningTime="2026-10-30 19:08:31.664909866 +0000 UTC m=+1906.775643150"
Oct 30 19:08:32.811313 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:32.811231    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-98a868"
Oct 30 19:08:32.844451 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:32.844423    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731d0671-acef-4e2f-8dd6-c7cca3ace772-kube-api-access-76ncp\" (UniqueName: \"kubernetes.io/projected/731d0671-acef-4e2f-8dd6-c7cca3ace772-kube-api-access-76ncp\") pod \"731d0671-acef-4e2f-8dd6-c7cca3ace772\" (UID: \"731d0671-acef-4e2f-8dd6-c7cca3ace772\") "
Oct 30 19:08:32.847154 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:32.847120    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731d0671-acef-4e2f-8dd6-c7cca3ace772-kube-api-access-76ncp" pod "731d0671-acef-4e2f-8dd6-c7cca3ace772" (UID: "731d0671-acef-4e2f-8dd6-c7cca3ace772"). InnerVolumeSpecName "kube-api-access-76ncp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:32.946107 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:32.946076    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76ncp\" (UniqueName: \"kubernetes.io/projected/731d0671-acef-4e2f-8dd6-c7cca3ace772-kube-api-access-76ncp\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:33.039611 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.037026    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731d0671-acef-4e2f-8dd6-c7cca3ace772" path="/var/lib/kubelet/pods/731d0671-acef-4e2f-8dd6-c7cca3ace772/volumes"
Oct 30 19:08:33.149878 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.149799    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8876-e851f1"]
Oct 30 19:08:33.155903 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.155874    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8876-e851f1"]
Oct 30 19:08:33.310799 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.310767    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8876-15fb45"]
Oct 30 19:08:33.311455 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.311441    4391 cpu_manager.go:414] "containerMap: removing container" podUID="6375b445-3df4-4d1a-be47-6650607365c9" containerName="test-curl-8876-e851f1"
Oct 30 19:08:33.311515 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.311456    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6375b445-3df4-4d1a-be47-6650607365c9" containerName="test-curl-8876-e851f1"
Oct 30 19:08:33.311605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.311591    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="6375b445-3df4-4d1a-be47-6650607365c9" containerName="test-curl-8876-e851f1"
Oct 30 19:08:33.315328 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.315307    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-15fb45"
Oct 30 19:08:33.351156 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.351127    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fml\" (UniqueName: \"kubernetes.io/projected/12721a43-049c-45c8-9e6d-c83d8ce1273c-kube-api-access-k2fml\") pod \"test-curl-8876-15fb45\" (UID: \"12721a43-049c-45c8-9e6d-c83d8ce1273c\") " pod="opendatahub/test-curl-8876-15fb45"
Oct 30 19:08:33.352253 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.352231    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-15fb45"]
Oct 30 19:08:33.452871 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.452837    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fml\" (UniqueName: \"kubernetes.io/projected/12721a43-049c-45c8-9e6d-c83d8ce1273c-kube-api-access-k2fml\") pod \"test-curl-8876-15fb45\" (UID: \"12721a43-049c-45c8-9e6d-c83d8ce1273c\") " pod="opendatahub/test-curl-8876-15fb45"
Oct 30 19:08:33.508049 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.508020    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fml\" (UniqueName: \"kubernetes.io/projected/12721a43-049c-45c8-9e6d-c83d8ce1273c-kube-api-access-k2fml\") pod \"test-curl-8876-15fb45\" (UID: \"12721a43-049c-45c8-9e6d-c83d8ce1273c\") " pod="opendatahub/test-curl-8876-15fb45"
Oct 30 19:08:33.570107 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.570071    4391 generic.go:356] "Generic (PLEG): container finished" podID="731d0671-acef-4e2f-8dd6-c7cca3ace772" containerID="0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4" exitCode=137
Oct 30 19:08:33.570107 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.570115    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-98a868"
Oct 30 19:08:33.570349 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.570167    4391 scope.go:122] "RemoveContainer" containerID="0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4"
Oct 30 19:08:33.570408 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.570367    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8876-e851f1" podUID="6375b445-3df4-4d1a-be47-6650607365c9" containerName="test-curl-8876-e851f1" containerID="cri-o://a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049" gracePeriod=2
Oct 30 19:08:33.580936 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.580918    4391 scope.go:122] "RemoveContainer" containerID="0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4"
Oct 30 19:08:33.581207 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:33.581190    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4\": container with ID starting with 0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4 not found: ID does not exist" containerID="0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4"
Oct 30 19:08:33.581260 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.581216    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4"} err="failed to get container status \"0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4\": rpc error: code = NotFound desc = could not find container \"0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4\": container with ID starting with 0f9724381050b2aea1e50917253fde50f52d9fba250800ff76a94679fd8188e4 not found: ID does not exist"
Oct 30 19:08:33.618643 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:33.618612    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-e851f1\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="6375b445-3df4-4d1a-be47-6650607365c9" pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:33.627997 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:33.627973    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-15fb45"
Oct 30 19:08:33.666489 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:33.666455    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-e851f1\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="6375b445-3df4-4d1a-be47-6650607365c9" pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:34.004004 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:34.003969    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8876-15fb45"]
Oct 30 19:08:34.004513 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:08:34.004489    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12721a43_049c_45c8_9e6d_c83d8ce1273c.slice/crio-42061b537e3fac8c5761a63d52b9efe7fc5e355d7de38a1d0e1789ec57dd162b WatchSource:0}: Error finding container 42061b537e3fac8c5761a63d52b9efe7fc5e355d7de38a1d0e1789ec57dd162b: Status 404 returned error can't find the container with id 42061b537e3fac8c5761a63d52b9efe7fc5e355d7de38a1d0e1789ec57dd162b
Oct 30 19:08:34.579776 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:34.579741    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-15fb45" event={"ID":"12721a43-049c-45c8-9e6d-c83d8ce1273c","Type":"ContainerStarted","Data":"6764fa808d54955464c8680319f31aa169f9b1e8c2ed6a2b71c417fd85bbc4fd"}
Oct 30 19:08:34.579776 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:34.579782    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8876-15fb45" event={"ID":"12721a43-049c-45c8-9e6d-c83d8ce1273c","Type":"ContainerStarted","Data":"42061b537e3fac8c5761a63d52b9efe7fc5e355d7de38a1d0e1789ec57dd162b"}
Oct 30 19:08:34.690735 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:34.690677    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8876-15fb45" podStartSLOduration=1.690664336 podStartE2EDuration="1.690664336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:34.687668459 +0000 UTC m=+1909.798401746" watchObservedRunningTime="2026-10-30 19:08:34.690664336 +0000 UTC m=+1909.801397619"
Oct 30 19:08:34.735634 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:34.735566    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-e851f1\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="6375b445-3df4-4d1a-be47-6650607365c9" pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:35.051293 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:35.051252    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-e851f1\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="6375b445-3df4-4d1a-be47-6650607365c9" pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:35.878685 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:35.878653    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:35.991368 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:35.991329    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6375b445-3df4-4d1a-be47-6650607365c9-kube-api-access-4xhtc\" (UniqueName: \"kubernetes.io/projected/6375b445-3df4-4d1a-be47-6650607365c9-kube-api-access-4xhtc\") pod \"6375b445-3df4-4d1a-be47-6650607365c9\" (UID: \"6375b445-3df4-4d1a-be47-6650607365c9\") "
Oct 30 19:08:35.993422 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:35.993388    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6375b445-3df4-4d1a-be47-6650607365c9-kube-api-access-4xhtc" pod "6375b445-3df4-4d1a-be47-6650607365c9" (UID: "6375b445-3df4-4d1a-be47-6650607365c9"). InnerVolumeSpecName "kube-api-access-4xhtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:36.094287 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:36.094192    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xhtc\" (UniqueName: \"kubernetes.io/projected/6375b445-3df4-4d1a-be47-6650607365c9-kube-api-access-4xhtc\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:36.223358 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:36.223325    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8876-15fb45"]
Oct 30 19:08:36.230797 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:36.230760    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8876-15fb45"]
Oct 30 19:08:36.599173 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:36.599140    4391 generic.go:356] "Generic (PLEG): container finished" podID="6375b445-3df4-4d1a-be47-6650607365c9" containerID="a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049" exitCode=137
Oct 30 19:08:36.599366 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:36.599180    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:36.599366 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:36.599239    4391 scope.go:122] "RemoveContainer" containerID="a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049"
Oct 30 19:08:36.599466 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:36.599448    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8876-15fb45" podUID="12721a43-049c-45c8-9e6d-c83d8ce1273c" containerName="test-curl-8876-15fb45" containerID="cri-o://6764fa808d54955464c8680319f31aa169f9b1e8c2ed6a2b71c417fd85bbc4fd" gracePeriod=2
Oct 30 19:08:36.610748 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:36.610728    4391 scope.go:122] "RemoveContainer" containerID="a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049"
Oct 30 19:08:36.611022 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:36.611002    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049\": container with ID starting with a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049 not found: ID does not exist" containerID="a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049"
Oct 30 19:08:36.611099 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:36.611029    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049"} err="failed to get container status \"a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049\": rpc error: code = NotFound desc = could not find container \"a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049\": container with ID starting with a07349f7a759d951c38999e93c9857bdaf2cd7c956cfe30a5d4f93999a83b049 not found: ID does not exist"
Oct 30 19:08:36.647438 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:36.647398    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-15fb45\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="12721a43-049c-45c8-9e6d-c83d8ce1273c" pod="opendatahub/test-curl-8876-15fb45"
Oct 30 19:08:36.695172 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:36.695132    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-15fb45\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="12721a43-049c-45c8-9e6d-c83d8ce1273c" pod="opendatahub/test-curl-8876-15fb45"
Oct 30 19:08:36.742880 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:36.742845    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8876-e851f1\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="6375b445-3df4-4d1a-be47-6650607365c9" pod="opendatahub/test-curl-8876-e851f1"
Oct 30 19:08:37.009118 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:37.009085    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6375b445-3df4-4d1a-be47-6650607365c9" path="/var/lib/kubelet/pods/6375b445-3df4-4d1a-be47-6650607365c9/volumes"
Oct 30 19:08:38.651379 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:38.651345    4391 generic.go:356] "Generic (PLEG): container finished" podID="12721a43-049c-45c8-9e6d-c83d8ce1273c" containerID="6764fa808d54955464c8680319f31aa169f9b1e8c2ed6a2b71c417fd85bbc4fd" exitCode=137
Oct 30 19:08:38.866264 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:38.866197    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-15fb45"
Oct 30 19:08:38.952030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:38.951998    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12721a43-049c-45c8-9e6d-c83d8ce1273c-kube-api-access-k2fml\" (UniqueName: \"kubernetes.io/projected/12721a43-049c-45c8-9e6d-c83d8ce1273c-kube-api-access-k2fml\") pod \"12721a43-049c-45c8-9e6d-c83d8ce1273c\" (UID: \"12721a43-049c-45c8-9e6d-c83d8ce1273c\") "
Oct 30 19:08:38.954170 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:38.954139    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12721a43-049c-45c8-9e6d-c83d8ce1273c-kube-api-access-k2fml" pod "12721a43-049c-45c8-9e6d-c83d8ce1273c" (UID: "12721a43-049c-45c8-9e6d-c83d8ce1273c"). InnerVolumeSpecName "kube-api-access-k2fml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:39.009688 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:39.009653    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12721a43-049c-45c8-9e6d-c83d8ce1273c" path="/var/lib/kubelet/pods/12721a43-049c-45c8-9e6d-c83d8ce1273c/volumes"
Oct 30 19:08:39.054572 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:39.054541    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2fml\" (UniqueName: \"kubernetes.io/projected/12721a43-049c-45c8-9e6d-c83d8ce1273c-kube-api-access-k2fml\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:39.660343 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:39.660312    4391 scope.go:122] "RemoveContainer" containerID="6764fa808d54955464c8680319f31aa169f9b1e8c2ed6a2b71c417fd85bbc4fd"
Oct 30 19:08:39.660793 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:39.660313    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8876-15fb45"
Oct 30 19:08:41.256313 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:08:41.256267    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:08:45.607156 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:08:45.607122    4391 scope.go:122] "RemoveContainer" containerID="ed8ee7f8eb639c61f65fa42a139e12f23e2cec80f4aa05fe87138e94cfeffa30"
Oct 30 19:09:01.560862 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:01.560776    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:09:11.253295 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:11.253248    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:09:12.856524 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.856490    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"]
Oct 30 19:09:12.857270 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.857252    4391 cpu_manager.go:414] "containerMap: removing container" podUID="12721a43-049c-45c8-9e6d-c83d8ce1273c" containerName="test-curl-8876-15fb45"
Oct 30 19:09:12.857314 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.857271    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12721a43-049c-45c8-9e6d-c83d8ce1273c" containerName="test-curl-8876-15fb45"
Oct 30 19:09:12.857418 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.857408    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="12721a43-049c-45c8-9e6d-c83d8ce1273c" containerName="test-curl-8876-15fb45"
Oct 30 19:09:12.861197 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.861178    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:12.918534 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.918495    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50829478-c347-4223-96eb-63902b988f15-openshift-service-ca\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:12.918830 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.918800    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-maas-api-tls\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:12.919176 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.918845    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkh2l\" (UniqueName: \"kubernetes.io/projected/50829478-c347-4223-96eb-63902b988f15-kube-api-access-xkh2l\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:12.919176 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.918923    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-metrics-certs\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:12.958470 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.958437    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cfb433-serving-cert\""
Oct 30 19:09:12.962876 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:12.962845    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"]
Oct 30 19:09:13.020246 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.020211    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50829478-c347-4223-96eb-63902b988f15-openshift-service-ca\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:13.020454 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.020266    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-maas-api-tls\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:13.020454 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:13.020433    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-serving-cert: secret "maas-api-e2e-api-a-cfb433-serving-cert" not found
Oct 30 19:09:13.020623 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:13.020493    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-maas-api-tls podName:50829478-c347-4223-96eb-63902b988f15 nodeName:}" failed. No retries permitted until 2026-10-30 19:09:13.520476838 +0000 UTC m=+1948.631210099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-maas-api-tls") pod "maas-api-e2e-api-a-cfb433-657b9469-zl59j" (UID: "50829478-c347-4223-96eb-63902b988f15") : secret "maas-api-e2e-api-a-cfb433-serving-cert" not found
Oct 30 19:09:13.020623 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.020439    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkh2l\" (UniqueName: \"kubernetes.io/projected/50829478-c347-4223-96eb-63902b988f15-kube-api-access-xkh2l\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:13.020623 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.020559    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-metrics-certs\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:13.021008 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.020985    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50829478-c347-4223-96eb-63902b988f15-openshift-service-ca\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:13.023042 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.023019    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-metrics-certs\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:13.075673 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.075635    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkh2l\" (UniqueName: \"kubernetes.io/projected/50829478-c347-4223-96eb-63902b988f15-kube-api-access-xkh2l\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:13.533898 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.533864    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-maas-api-tls\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:13.536266 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.536241    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-maas-api-tls\") pod \"maas-api-e2e-api-a-cfb433-657b9469-zl59j\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:13.577567 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.577535    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"]
Oct 30 19:09:13.577795 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.577770    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy" containerID="cri-o://b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309" gracePeriod=30
Oct 30 19:09:13.632524 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.632480    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"]
Oct 30 19:09:13.633049 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.633005    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy" containerID="cri-o://20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c" gracePeriod=30
Oct 30 19:09:13.638455 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.638431    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"]
Oct 30 19:09:13.638704 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.638670    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy" containerID="cri-o://3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53" gracePeriod=30
Oct 30 19:09:13.775936 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.775899    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:13.784375 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.784304    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"]
Oct 30 19:09:13.784627 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.784560    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy" containerID="cri-o://8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a" gracePeriod=30
Oct 30 19:09:13.869085 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.869055    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"]
Oct 30 19:09:13.869762 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.869736    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy" containerID="cri-o://1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c" gracePeriod=30
Oct 30 19:09:13.915115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.915084    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"]
Oct 30 19:09:13.915838 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:13.915810    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy" containerID="cri-o://5eb2d90d0c0a223db16b9c62d0c6470ef7df4a0328eee63149cc09985db76d87" gracePeriod=30
Oct 30 19:09:14.184559 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:09:14.184526    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50829478_c347_4223_96eb_63902b988f15.slice/crio-37a3ef628b0e71b989f7033a01155656141b7974ee28093be6461155d64838ef WatchSource:0}: Error finding container 37a3ef628b0e71b989f7033a01155656141b7974ee28093be6461155d64838ef: Status 404 returned error can't find the container with id 37a3ef628b0e71b989f7033a01155656141b7974ee28093be6461155d64838ef
Oct 30 19:09:14.184987 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:14.184960    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"]
Oct 30 19:09:14.202059 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:14.202027    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j" event={"ID":"50829478-c347-4223-96eb-63902b988f15","Type":"ContainerStarted","Data":"37a3ef628b0e71b989f7033a01155656141b7974ee28093be6461155d64838ef"}
Oct 30 19:09:15.212097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:15.211220    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j" event={"ID":"50829478-c347-4223-96eb-63902b988f15","Type":"ContainerStarted","Data":"bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510"}
Oct 30 19:09:15.212097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:15.211355    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:15.327380 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:15.327259    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j" podStartSLOduration=2.470534889 podStartE2EDuration="3.32723988s" totalImagesPullingTime="856.704991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:15.323002105 +0000 UTC m=+1950.433735380" watchObservedRunningTime="2026-10-30 19:09:15.32723988 +0000 UTC m=+1950.437973165"
Oct 30 19:09:18.042469 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:18.042433    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:18.042939 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:18.042509    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:18.618545 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:18.618516    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:18.618705 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:18.618567    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:18.620224 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:18.620200    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:18.620306 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:18.620261    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:18.833952 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:18.833647    4391 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:18.833952 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:18.833704    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:19.057903 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.057869    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:09:19.080607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.078831    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:09:19.085700 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.081778    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 19:09:19.127040 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.127014    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 19:09:19.161091 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161019    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37e1504c-56bb-474d-8878-fe512509d229-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37e1504c-56bb-474d-8878-fe512509d229-istiod-ca-cert\") pod \"37e1504c-56bb-474d-8878-fe512509d229\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") "
Oct 30 19:09:19.161091 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161065    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-socket\") pod \"37e1504c-56bb-474d-8878-fe512509d229\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") "
Oct 30 19:09:19.161296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161093    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-credential-socket\") pod \"e2aeabe1-d147-468a-855f-dbb4573c0d33\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") "
Oct 30 19:09:19.161296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161137    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-credential-socket\") pod \"294c1b7d-6184-49db-9b0d-4c4ab690a647\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") "
Oct 30 19:09:19.161296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161183    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-kube-api-access-zmhcl\" (UniqueName: \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-kube-api-access-zmhcl\") pod \"37e1504c-56bb-474d-8878-fe512509d229\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") "
Oct 30 19:09:19.161296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161237    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-podinfo\") pod \"294c1b7d-6184-49db-9b0d-4c4ab690a647\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") "
Oct 30 19:09:19.161296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161281    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-istio-token\" (UniqueName: \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-istio-token\") pod \"37e1504c-56bb-474d-8878-fe512509d229\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") "
Oct 30 19:09:19.161544 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161331    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-socket\") pod \"e2aeabe1-d147-468a-855f-dbb4573c0d33\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") "
Oct 30 19:09:19.161544 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161391    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-credential-socket" pod "e2aeabe1-d147-468a-855f-dbb4573c0d33" (UID: "e2aeabe1-d147-468a-855f-dbb4573c0d33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.161544 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161432    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e1504c-56bb-474d-8878-fe512509d229-istiod-ca-cert" pod "37e1504c-56bb-474d-8878-fe512509d229" (UID: "37e1504c-56bb-474d-8878-fe512509d229"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.161544 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161448    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-data\") pod \"294c1b7d-6184-49db-9b0d-4c4ab690a647\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") "
Oct 30 19:09:19.161544 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161477    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-socket\") pod \"294c1b7d-6184-49db-9b0d-4c4ab690a647\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") "
Oct 30 19:09:19.161544 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161507    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-kube-api-access-8rdlq\" (UniqueName: \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-kube-api-access-8rdlq\") pod \"e2aeabe1-d147-468a-855f-dbb4573c0d33\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") "
Oct 30 19:09:19.161872 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161549    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-envoy\") pod \"294c1b7d-6184-49db-9b0d-4c4ab690a647\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") "
Oct 30 19:09:19.161872 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161622    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37e1504c-56bb-474d-8878-fe512509d229-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37e1504c-56bb-474d-8878-fe512509d229-istio-podinfo\") pod \"37e1504c-56bb-474d-8878-fe512509d229\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") "
Oct 30 19:09:19.161872 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161666    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2aeabe1-d147-468a-855f-dbb4573c0d33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2aeabe1-d147-468a-855f-dbb4573c0d33-istiod-ca-cert\") pod \"e2aeabe1-d147-468a-855f-dbb4573c0d33\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") "
Oct 30 19:09:19.161872 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161733    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-certs\") pod \"e2aeabe1-d147-468a-855f-dbb4573c0d33\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") "
Oct 30 19:09:19.161872 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161760    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-credential-socket\") pod \"37e1504c-56bb-474d-8878-fe512509d229\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") "
Oct 30 19:09:19.161872 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161794    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-kube-api-access-tjv2b\" (UniqueName: \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-kube-api-access-tjv2b\") pod \"294c1b7d-6184-49db-9b0d-4c4ab690a647\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") "
Oct 30 19:09:19.161872 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161829    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294c1b7d-6184-49db-9b0d-4c4ab690a647-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294c1b7d-6184-49db-9b0d-4c4ab690a647-istiod-ca-cert\") pod \"294c1b7d-6184-49db-9b0d-4c4ab690a647\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") "
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161880    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-socket" pod "37e1504c-56bb-474d-8878-fe512509d229" (UID: "37e1504c-56bb-474d-8878-fe512509d229"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161911    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-token\" (UniqueName: \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-token\") pod \"e2aeabe1-d147-468a-855f-dbb4573c0d33\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") "
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161947    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-podinfo\") pod \"e2aeabe1-d147-468a-855f-dbb4573c0d33\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") "
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.161973    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-envoy\") pod \"e2aeabe1-d147-468a-855f-dbb4573c0d33\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") "
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.162007    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-certs\") pod \"37e1504c-56bb-474d-8878-fe512509d229\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") "
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.162017    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-socket" pod "294c1b7d-6184-49db-9b0d-4c4ab690a647" (UID: "294c1b7d-6184-49db-9b0d-4c4ab690a647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.162052    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-certs\") pod \"294c1b7d-6184-49db-9b0d-4c4ab690a647\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") "
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.162079    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-token\" (UniqueName: \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-token\") pod \"294c1b7d-6184-49db-9b0d-4c4ab690a647\" (UID: \"294c1b7d-6184-49db-9b0d-4c4ab690a647\") "
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.162144    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-data\") pod \"e2aeabe1-d147-468a-855f-dbb4573c0d33\" (UID: \"e2aeabe1-d147-468a-855f-dbb4573c0d33\") "
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.162220    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-socket" pod "e2aeabe1-d147-468a-855f-dbb4573c0d33" (UID: "e2aeabe1-d147-468a-855f-dbb4573c0d33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.162229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.162231    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-envoy\") pod \"37e1504c-56bb-474d-8878-fe512509d229\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") "
Oct 30 19:09:19.162769 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.162258    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-data\") pod \"37e1504c-56bb-474d-8878-fe512509d229\" (UID: \"37e1504c-56bb-474d-8878-fe512509d229\") "
Oct 30 19:09:19.162769 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.162437    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-data" pod "294c1b7d-6184-49db-9b0d-4c4ab690a647" (UID: "294c1b7d-6184-49db-9b0d-4c4ab690a647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.162769 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.162651    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-data" pod "37e1504c-56bb-474d-8878-fe512509d229" (UID: "37e1504c-56bb-474d-8878-fe512509d229"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.163142 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.163123    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.163200 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.163147    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.163200 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.163161    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.163200 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.163173    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.163200 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.163189    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37e1504c-56bb-474d-8878-fe512509d229-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.163394 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.163201    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.163394 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.163216    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.165808 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.165592    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-kube-api-access-8rdlq" pod "e2aeabe1-d147-468a-855f-dbb4573c0d33" (UID: "e2aeabe1-d147-468a-855f-dbb4573c0d33"). InnerVolumeSpecName "kube-api-access-8rdlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.165950 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.165838    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-token" pod "e2aeabe1-d147-468a-855f-dbb4573c0d33" (UID: "e2aeabe1-d147-468a-855f-dbb4573c0d33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.166360 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.166037    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-credential-socket" pod "37e1504c-56bb-474d-8878-fe512509d229" (UID: "37e1504c-56bb-474d-8878-fe512509d229"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.166360 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.166089    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-certs" pod "e2aeabe1-d147-468a-855f-dbb4573c0d33" (UID: "e2aeabe1-d147-468a-855f-dbb4573c0d33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.166891 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.166651    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-envoy" pod "294c1b7d-6184-49db-9b0d-4c4ab690a647" (UID: "294c1b7d-6184-49db-9b0d-4c4ab690a647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.167401 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.167161    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2aeabe1-d147-468a-855f-dbb4573c0d33-istiod-ca-cert" pod "e2aeabe1-d147-468a-855f-dbb4573c0d33" (UID: "e2aeabe1-d147-468a-855f-dbb4573c0d33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.167887 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.167677    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-istio-token" pod "37e1504c-56bb-474d-8878-fe512509d229" (UID: "37e1504c-56bb-474d-8878-fe512509d229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.167887 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.167782    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-podinfo" pod "e2aeabe1-d147-468a-855f-dbb4573c0d33" (UID: "e2aeabe1-d147-468a-855f-dbb4573c0d33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.168269 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.168087    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-data" pod "e2aeabe1-d147-468a-855f-dbb4573c0d33" (UID: "e2aeabe1-d147-468a-855f-dbb4573c0d33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.168643 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.168556    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-certs" pod "294c1b7d-6184-49db-9b0d-4c4ab690a647" (UID: "294c1b7d-6184-49db-9b0d-4c4ab690a647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.169087 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.169061    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-credential-socket" pod "294c1b7d-6184-49db-9b0d-4c4ab690a647" (UID: "294c1b7d-6184-49db-9b0d-4c4ab690a647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.169253 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.169235    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-certs" pod "37e1504c-56bb-474d-8878-fe512509d229" (UID: "37e1504c-56bb-474d-8878-fe512509d229"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.169402 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.169355    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-kube-api-access-zmhcl" pod "37e1504c-56bb-474d-8878-fe512509d229" (UID: "37e1504c-56bb-474d-8878-fe512509d229"). InnerVolumeSpecName "kube-api-access-zmhcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.169527 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.169509    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294c1b7d-6184-49db-9b0d-4c4ab690a647-istiod-ca-cert" pod "294c1b7d-6184-49db-9b0d-4c4ab690a647" (UID: "294c1b7d-6184-49db-9b0d-4c4ab690a647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.170183 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.170157    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-kube-api-access-tjv2b" pod "294c1b7d-6184-49db-9b0d-4c4ab690a647" (UID: "294c1b7d-6184-49db-9b0d-4c4ab690a647"). InnerVolumeSpecName "kube-api-access-tjv2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.170469 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.170449    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37e1504c-56bb-474d-8878-fe512509d229-istio-podinfo" pod "37e1504c-56bb-474d-8878-fe512509d229" (UID: "37e1504c-56bb-474d-8878-fe512509d229"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.170736 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.170711    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-podinfo" pod "294c1b7d-6184-49db-9b0d-4c4ab690a647" (UID: "294c1b7d-6184-49db-9b0d-4c4ab690a647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.172150 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.172121    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-envoy" pod "37e1504c-56bb-474d-8878-fe512509d229" (UID: "37e1504c-56bb-474d-8878-fe512509d229"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.172825 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.172791    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-token" pod "294c1b7d-6184-49db-9b0d-4c4ab690a647" (UID: "294c1b7d-6184-49db-9b0d-4c4ab690a647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.192141 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.192072    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-envoy" pod "e2aeabe1-d147-468a-855f-dbb4573c0d33" (UID: "e2aeabe1-d147-468a-855f-dbb4573c0d33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.232812 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.232760    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 19:09:19.244332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.244297    4391 generic.go:356] "Generic (PLEG): container finished" podID="37e1504c-56bb-474d-8878-fe512509d229" containerID="b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309" exitCode=0
Oct 30 19:09:19.244503 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.244439    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" event={"ID":"37e1504c-56bb-474d-8878-fe512509d229","Type":"ContainerDied","Data":"b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309"}
Oct 30 19:09:19.244503 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.244490    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf" event={"ID":"37e1504c-56bb-474d-8878-fe512509d229","Type":"ContainerDied","Data":"225bddca29502136c91813161d456e3f15a16b01507456c15ebb3ba3a40b6a2f"}
Oct 30 19:09:19.244639 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.244518    4391 scope.go:122] "RemoveContainer" containerID="b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309"
Oct 30 19:09:19.244695 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.244677    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"
Oct 30 19:09:19.257966 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.257898    4391 generic.go:356] "Generic (PLEG): container finished" podID="17b31304-1ee2-4030-88a1-525f32125a43" containerID="8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a" exitCode=0
Oct 30 19:09:19.258100 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.257979    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" event={"ID":"17b31304-1ee2-4030-88a1-525f32125a43","Type":"ContainerDied","Data":"8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a"}
Oct 30 19:09:19.258100 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.258045    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9" event={"ID":"17b31304-1ee2-4030-88a1-525f32125a43","Type":"ContainerDied","Data":"fd749b24491f61ba51298a87295aa376bf34e6ded6a7687b9e06d970b86e4d57"}
Oct 30 19:09:19.258100 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.258086    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"
Oct 30 19:09:19.260887 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.260847    4391 generic.go:356] "Generic (PLEG): container finished" podID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerID="3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53" exitCode=0
Oct 30 19:09:19.261018 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.261009    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"
Oct 30 19:09:19.261207 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.261166    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" event={"ID":"e2aeabe1-d147-468a-855f-dbb4573c0d33","Type":"ContainerDied","Data":"3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53"}
Oct 30 19:09:19.261392 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.261372    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989" event={"ID":"e2aeabe1-d147-468a-855f-dbb4573c0d33","Type":"ContainerDied","Data":"42f41ded44549ba1cab55232005f7cde7cb44c67f14c5342979654465bab205c"}
Oct 30 19:09:19.263051 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.262983    4391 generic.go:356] "Generic (PLEG): container finished" podID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerID="20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c" exitCode=0
Oct 30 19:09:19.263051 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.263030    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"
Oct 30 19:09:19.263251 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.263105    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" event={"ID":"294c1b7d-6184-49db-9b0d-4c4ab690a647","Type":"ContainerDied","Data":"20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c"}
Oct 30 19:09:19.263251 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.263146    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s" event={"ID":"294c1b7d-6184-49db-9b0d-4c4ab690a647","Type":"ContainerDied","Data":"a3725424fc37001728db27067cafbd449cec30f34aa3660fe9c81c7ed433781c"}
Oct 30 19:09:19.264333 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.264313    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-istio-token\" (UniqueName: \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-istio-token\") pod \"17b31304-1ee2-4030-88a1-525f32125a43\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") "
Oct 30 19:09:19.264450 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.264367    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-certs\") pod \"17b31304-1ee2-4030-88a1-525f32125a43\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") "
Oct 30 19:09:19.264515 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.264462    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-credential-socket\") pod \"17b31304-1ee2-4030-88a1-525f32125a43\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") "
Oct 30 19:09:19.264515 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.264504    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-socket\") pod \"17b31304-1ee2-4030-88a1-525f32125a43\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") "
Oct 30 19:09:19.264640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.264547    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17b31304-1ee2-4030-88a1-525f32125a43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b31304-1ee2-4030-88a1-525f32125a43-istiod-ca-cert\") pod \"17b31304-1ee2-4030-88a1-525f32125a43\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") "
Oct 30 19:09:19.264730 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.264690    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-kube-api-access-lgz7g\" (UniqueName: \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-kube-api-access-lgz7g\") pod \"17b31304-1ee2-4030-88a1-525f32125a43\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") "
Oct 30 19:09:19.264788 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.264752    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17b31304-1ee2-4030-88a1-525f32125a43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b31304-1ee2-4030-88a1-525f32125a43-istio-podinfo\") pod \"17b31304-1ee2-4030-88a1-525f32125a43\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") "
Oct 30 19:09:19.264921 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.264907    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-envoy\") pod \"17b31304-1ee2-4030-88a1-525f32125a43\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") "
Oct 30 19:09:19.264985 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.264972    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-data\") pod \"17b31304-1ee2-4030-88a1-525f32125a43\" (UID: \"17b31304-1ee2-4030-88a1-525f32125a43\") "
Oct 30 19:09:19.265450 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265433    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.265540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265453    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.265540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265466    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rdlq\" (UniqueName: \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-kube-api-access-8rdlq\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.265540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265478    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.265540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265491    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37e1504c-56bb-474d-8878-fe512509d229-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.265540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265504    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2aeabe1-d147-468a-855f-dbb4573c0d33-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.265540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265517    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.265540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265531    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265544    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjv2b\" (UniqueName: \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-kube-api-access-tjv2b\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265558    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/294c1b7d-6184-49db-9b0d-4c4ab690a647-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265572    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265604    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265617    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265631    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265644    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265659    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/294c1b7d-6184-49db-9b0d-4c4ab690a647-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265672    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2aeabe1-d147-468a-855f-dbb4573c0d33-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265685    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37e1504c-56bb-474d-8878-fe512509d229-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265727    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/294c1b7d-6184-49db-9b0d-4c4ab690a647-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.266030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.265743    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmhcl\" (UniqueName: \"kubernetes.io/projected/37e1504c-56bb-474d-8878-fe512509d229-kube-api-access-zmhcl\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.267801 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.267785    4391 scope.go:122] "RemoveContainer" containerID="b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309"
Oct 30 19:09:19.268697 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.268100    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-socket" pod "17b31304-1ee2-4030-88a1-525f32125a43" (UID: "17b31304-1ee2-4030-88a1-525f32125a43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.268810 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.268155    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-data" pod "17b31304-1ee2-4030-88a1-525f32125a43" (UID: "17b31304-1ee2-4030-88a1-525f32125a43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.268905 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.268469    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b31304-1ee2-4030-88a1-525f32125a43-istiod-ca-cert" pod "17b31304-1ee2-4030-88a1-525f32125a43" (UID: "17b31304-1ee2-4030-88a1-525f32125a43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.268989 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.268641    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-certs" pod "17b31304-1ee2-4030-88a1-525f32125a43" (UID: "17b31304-1ee2-4030-88a1-525f32125a43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.269069 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.268978    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-credential-socket" pod "17b31304-1ee2-4030-88a1-525f32125a43" (UID: "17b31304-1ee2-4030-88a1-525f32125a43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.269149 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:19.269057    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309\": container with ID starting with b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309 not found: ID does not exist" containerID="b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309"
Oct 30 19:09:19.269149 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.269093    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309"} err="failed to get container status \"b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309\": rpc error: code = NotFound desc = could not find container \"b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309\": container with ID starting with b29c8892714308c4255e04e53b57f1e94feb60bbb99a9c8ac8011b0d918f9309 not found: ID does not exist"
Oct 30 19:09:19.269149 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.269116    4391 scope.go:122] "RemoveContainer" containerID="8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a"
Oct 30 19:09:19.270096 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.270070    4391 generic.go:356] "Generic (PLEG): container finished" podID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerID="1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c" exitCode=0
Oct 30 19:09:19.270709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.270190    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"
Oct 30 19:09:19.270709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.270319    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" event={"ID":"3d90d268-ee2d-49ac-9634-eaa512441be9","Type":"ContainerDied","Data":"1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c"}
Oct 30 19:09:19.270709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.270354    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt" event={"ID":"3d90d268-ee2d-49ac-9634-eaa512441be9","Type":"ContainerDied","Data":"4607ffcc87d2744f377d4800651ac91cfd00d4b9cb6fc6eb51a3c2014337c9c4"}
Oct 30 19:09:19.274311 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.274285    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-kube-api-access-lgz7g" pod "17b31304-1ee2-4030-88a1-525f32125a43" (UID: "17b31304-1ee2-4030-88a1-525f32125a43"). InnerVolumeSpecName "kube-api-access-lgz7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.277076 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.277052    4391 generic.go:356] "Generic (PLEG): container finished" podID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerID="5eb2d90d0c0a223db16b9c62d0c6470ef7df4a0328eee63149cc09985db76d87" exitCode=0
Oct 30 19:09:19.277192 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.277122    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" event={"ID":"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f","Type":"ContainerDied","Data":"5eb2d90d0c0a223db16b9c62d0c6470ef7df4a0328eee63149cc09985db76d87"}
Oct 30 19:09:19.280704 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.280684    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:09:19.281144 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.281114    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17b31304-1ee2-4030-88a1-525f32125a43-istio-podinfo" pod "17b31304-1ee2-4030-88a1-525f32125a43" (UID: "17b31304-1ee2-4030-88a1-525f32125a43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.281230 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.281169    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-envoy" pod "17b31304-1ee2-4030-88a1-525f32125a43" (UID: "17b31304-1ee2-4030-88a1-525f32125a43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.281230 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.281171    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-istio-token" pod "17b31304-1ee2-4030-88a1-525f32125a43" (UID: "17b31304-1ee2-4030-88a1-525f32125a43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.286233 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:19.286202    4391 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e1504c_56bb_474d_8878_fe512509d229.slice/crio-225bddca29502136c91813161d456e3f15a16b01507456c15ebb3ba3a40b6a2f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294c1b7d_6184_49db_9b0d_4c4ab690a647.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:09:19.288803 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.288774    4391 scope.go:122] "RemoveContainer" containerID="8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a"
Oct 30 19:09:19.289186 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:19.289136    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a\": container with ID starting with 8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a not found: ID does not exist" containerID="8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a"
Oct 30 19:09:19.289186 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.289172    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a"} err="failed to get container status \"8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a\": rpc error: code = NotFound desc = could not find container \"8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a\": container with ID starting with 8d7eafe448c9cbb8d2c6759d0adff159928971b0eb538a72b927f346e401143a not found: ID does not exist"
Oct 30 19:09:19.289349 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.289193    4391 scope.go:122] "RemoveContainer" containerID="3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53"
Oct 30 19:09:19.301502 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.301483    4391 scope.go:122] "RemoveContainer" containerID="3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53"
Oct 30 19:09:19.301816 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:19.301792    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53\": container with ID starting with 3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53 not found: ID does not exist" containerID="3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53"
Oct 30 19:09:19.301901 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.301825    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53"} err="failed to get container status \"3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53\": rpc error: code = NotFound desc = could not find container \"3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53\": container with ID starting with 3240f69f6c8def915976dbaa4a7f16979860ee0da8d2a3c68bf9cf7991e29f53 not found: ID does not exist"
Oct 30 19:09:19.301901 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.301848    4391 scope.go:122] "RemoveContainer" containerID="20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c"
Oct 30 19:09:19.313973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.313948    4391 scope.go:122] "RemoveContainer" containerID="20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c"
Oct 30 19:09:19.314326 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:19.314305    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c\": container with ID starting with 20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c not found: ID does not exist" containerID="20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c"
Oct 30 19:09:19.314406 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.314333    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c"} err="failed to get container status \"20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c\": rpc error: code = NotFound desc = could not find container \"20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c\": container with ID starting with 20938e16381ebbb61504f1f840e13abb4462c9da2c8e3b6654238d461cb3dd5c not found: ID does not exist"
Oct 30 19:09:19.314406 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.314351    4391 scope.go:122] "RemoveContainer" containerID="1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c"
Oct 30 19:09:19.324047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.324029    4391 scope.go:122] "RemoveContainer" containerID="1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c"
Oct 30 19:09:19.324385 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:19.324351    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c\": container with ID starting with 1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c not found: ID does not exist" containerID="1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c"
Oct 30 19:09:19.324456 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.324394    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c"} err="failed to get container status \"1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c\": rpc error: code = NotFound desc = could not find container \"1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c\": container with ID starting with 1dd54a0a5c70f546379c97676b8147cdcd7e7323e9b2a3289656a026128ca47c not found: ID does not exist"
Oct 30 19:09:19.366332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366289    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-podinfo\") pod \"3d90d268-ee2d-49ac-9634-eaa512441be9\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") "
Oct 30 19:09:19.366499 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366354    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-credential-socket\") pod \"3d90d268-ee2d-49ac-9634-eaa512441be9\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") "
Oct 30 19:09:19.366499 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366413    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-credential-socket\") pod \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") "
Oct 30 19:09:19.366693 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366500    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-kube-api-access-2zgcm\" (UniqueName: \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-kube-api-access-2zgcm\") pod \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") "
Oct 30 19:09:19.366693 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366539    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d90d268-ee2d-49ac-9634-eaa512441be9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d90d268-ee2d-49ac-9634-eaa512441be9-istiod-ca-cert\") pod \"3d90d268-ee2d-49ac-9634-eaa512441be9\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") "
Oct 30 19:09:19.366693 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366686    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-envoy\") pod \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") "
Oct 30 19:09:19.366855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366721    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-token\" (UniqueName: \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-token\") pod \"3d90d268-ee2d-49ac-9634-eaa512441be9\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") "
Oct 30 19:09:19.366855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366795    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-podinfo\") pod \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") "
Oct 30 19:09:19.366855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366794    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-credential-socket" pod "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" (UID: "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.366855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366803    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-credential-socket" pod "3d90d268-ee2d-49ac-9634-eaa512441be9" (UID: "3d90d268-ee2d-49ac-9634-eaa512441be9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.366855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366824    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istiod-ca-cert\") pod \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") "
Oct 30 19:09:19.367112 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.366990    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d90d268-ee2d-49ac-9634-eaa512441be9-istiod-ca-cert" pod "3d90d268-ee2d-49ac-9634-eaa512441be9" (UID: "3d90d268-ee2d-49ac-9634-eaa512441be9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.367112 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367021    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-kube-api-access-cr56b\" (UniqueName: \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-kube-api-access-cr56b\") pod \"3d90d268-ee2d-49ac-9634-eaa512441be9\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") "
Oct 30 19:09:19.367112 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367068    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-socket\") pod \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") "
Oct 30 19:09:19.367112 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367087    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-data\") pod \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") "
Oct 30 19:09:19.367112 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367106    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-socket\") pod \"3d90d268-ee2d-49ac-9634-eaa512441be9\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") "
Oct 30 19:09:19.367370 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367161    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-token\" (UniqueName: \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-token\") pod \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") "
Oct 30 19:09:19.367370 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367182    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-envoy\") pod \"3d90d268-ee2d-49ac-9634-eaa512441be9\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") "
Oct 30 19:09:19.367370 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367215    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-certs\") pod \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\" (UID: \"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f\") "
Oct 30 19:09:19.367370 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367293    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-data\") pod \"3d90d268-ee2d-49ac-9634-eaa512441be9\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") "
Oct 30 19:09:19.367370 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367313    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-certs\") pod \"3d90d268-ee2d-49ac-9634-eaa512441be9\" (UID: \"3d90d268-ee2d-49ac-9634-eaa512441be9\") "
Oct 30 19:09:19.367795 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367778    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367795 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367796    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367805    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367815    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367825    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d90d268-ee2d-49ac-9634-eaa512441be9-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367834    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367842    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367850    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367859    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b31304-1ee2-4030-88a1-525f32125a43-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367867    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b31304-1ee2-4030-88a1-525f32125a43-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367875    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgz7g\" (UniqueName: \"kubernetes.io/projected/17b31304-1ee2-4030-88a1-525f32125a43-kube-api-access-lgz7g\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.367925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.367884    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b31304-1ee2-4030-88a1-525f32125a43-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.368428 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.368378    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-socket" pod "3d90d268-ee2d-49ac-9634-eaa512441be9" (UID: "3d90d268-ee2d-49ac-9634-eaa512441be9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.368642 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.368619    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istiod-ca-cert" pod "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" (UID: "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.368919 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.368728    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-certs" pod "3d90d268-ee2d-49ac-9634-eaa512441be9" (UID: "3d90d268-ee2d-49ac-9634-eaa512441be9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.369263 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.369090    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-certs" pod "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" (UID: "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.369263 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.369147    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-token" pod "3d90d268-ee2d-49ac-9634-eaa512441be9" (UID: "3d90d268-ee2d-49ac-9634-eaa512441be9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.369648 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.369449    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-data" pod "3d90d268-ee2d-49ac-9634-eaa512441be9" (UID: "3d90d268-ee2d-49ac-9634-eaa512441be9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.369966 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.369781    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-socket" pod "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" (UID: "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.369966 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.369788    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-podinfo" pod "3d90d268-ee2d-49ac-9634-eaa512441be9" (UID: "3d90d268-ee2d-49ac-9634-eaa512441be9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.370178 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.370146    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-data" pod "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" (UID: "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.371320 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.371299    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-podinfo" pod "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" (UID: "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.371417 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.371335    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-kube-api-access-cr56b" pod "3d90d268-ee2d-49ac-9634-eaa512441be9" (UID: "3d90d268-ee2d-49ac-9634-eaa512441be9"). InnerVolumeSpecName "kube-api-access-cr56b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.371417 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.371376    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-envoy" pod "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" (UID: "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.371552 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.371504    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-kube-api-access-2zgcm" pod "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" (UID: "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f"). InnerVolumeSpecName "kube-api-access-2zgcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.372864 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.372843    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-token" pod "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" (UID: "5eba9a14-88d6-4532-9ad1-14e3f1b4d99f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.372864 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.372846    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-envoy" pod "3d90d268-ee2d-49ac-9634-eaa512441be9" (UID: "3d90d268-ee2d-49ac-9634-eaa512441be9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.469838 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469745    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.469838 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469789    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.469838 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469804    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.469838 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469819    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.469838 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469835    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zgcm\" (UniqueName: \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-kube-api-access-2zgcm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.470222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469850    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.470222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469862    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.470222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469873    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.470222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469885    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.470222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469897    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr56b\" (UniqueName: \"kubernetes.io/projected/3d90d268-ee2d-49ac-9634-eaa512441be9-kube-api-access-cr56b\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.470222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469910    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.470222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469929    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.470222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469942    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.470222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469956    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.470222 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.469968    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d90d268-ee2d-49ac-9634-eaa512441be9-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.672385 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.672348    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"]
Oct 30 19:09:19.676431 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.676400    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-2x85s"]
Oct 30 19:09:19.784640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.784530    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"]
Oct 30 19:09:19.788551 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.788525    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-g6pcf"]
Oct 30 19:09:19.893729 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.893676    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"]
Oct 30 19:09:19.897865 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:19.897835    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-d2989"]
Oct 30 19:09:20.299798 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:20.299760    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"
Oct 30 19:09:20.300262 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:20.299803    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8" event={"ID":"5eba9a14-88d6-4532-9ad1-14e3f1b4d99f","Type":"ContainerDied","Data":"20f253a65e2582e7007b54611e4cadb9a61d2fb0cd636c0dc28b77ade7c212cb"}
Oct 30 19:09:20.300262 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:20.299857    4391 scope.go:122] "RemoveContainer" containerID="5eb2d90d0c0a223db16b9c62d0c6470ef7df4a0328eee63149cc09985db76d87"
Oct 30 19:09:20.301727 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:20.301635    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"]
Oct 30 19:09:20.305759 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:20.305735    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-ls7vt"]
Oct 30 19:09:20.410172 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:20.410139    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"]
Oct 30 19:09:20.414538 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:20.414505    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-gwsf9"]
Oct 30 19:09:20.524216 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:20.524177    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"]
Oct 30 19:09:20.529297 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:20.529263    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-bskg8"]
Oct 30 19:09:21.007839 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:21.007799    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b31304-1ee2-4030-88a1-525f32125a43" path="/var/lib/kubelet/pods/17b31304-1ee2-4030-88a1-525f32125a43/volumes"
Oct 30 19:09:21.008332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:21.008316    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" path="/var/lib/kubelet/pods/294c1b7d-6184-49db-9b0d-4c4ab690a647/volumes"
Oct 30 19:09:21.008819 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:21.008801    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37e1504c-56bb-474d-8878-fe512509d229" path="/var/lib/kubelet/pods/37e1504c-56bb-474d-8878-fe512509d229/volumes"
Oct 30 19:09:21.009607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:21.009566    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" path="/var/lib/kubelet/pods/3d90d268-ee2d-49ac-9634-eaa512441be9/volumes"
Oct 30 19:09:21.010031 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:21.010017    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" path="/var/lib/kubelet/pods/5eba9a14-88d6-4532-9ad1-14e3f1b4d99f/volumes"
Oct 30 19:09:21.010450 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:21.010438    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" path="/var/lib/kubelet/pods/e2aeabe1-d147-468a-855f-dbb4573c0d33/volumes"
Oct 30 19:09:21.223882 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:21.223853    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:09:23.038598 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.038540    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"]
Oct 30 19:09:23.039210 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039194    4391 cpu_manager.go:414] "containerMap: removing container" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy"
Oct 30 19:09:23.039265 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039211    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy"
Oct 30 19:09:23.039265 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039228    4391 cpu_manager.go:414] "containerMap: removing container" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy"
Oct 30 19:09:23.039265 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039233    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy"
Oct 30 19:09:23.039265 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039243    4391 cpu_manager.go:414] "containerMap: removing container" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy"
Oct 30 19:09:23.039265 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039248    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy"
Oct 30 19:09:23.039429 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039273    4391 cpu_manager.go:414] "containerMap: removing container" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy"
Oct 30 19:09:23.039429 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039278    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy"
Oct 30 19:09:23.039429 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039292    4391 cpu_manager.go:414] "containerMap: removing container" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy"
Oct 30 19:09:23.039429 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039301    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy"
Oct 30 19:09:23.039429 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039321    4391 cpu_manager.go:414] "containerMap: removing container" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy"
Oct 30 19:09:23.039429 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039326    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy"
Oct 30 19:09:23.039732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039460    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b31304-1ee2-4030-88a1-525f32125a43" containerName="istio-proxy"
Oct 30 19:09:23.039732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039477    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eba9a14-88d6-4532-9ad1-14e3f1b4d99f" containerName="istio-proxy"
Oct 30 19:09:23.039732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039491    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="37e1504c-56bb-474d-8878-fe512509d229" containerName="istio-proxy"
Oct 30 19:09:23.039732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039497    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="294c1b7d-6184-49db-9b0d-4c4ab690a647" containerName="istio-proxy"
Oct 30 19:09:23.039732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039504    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2aeabe1-d147-468a-855f-dbb4573c0d33" containerName="istio-proxy"
Oct 30 19:09:23.039732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.039526    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d90d268-ee2d-49ac-9634-eaa512441be9" containerName="istio-proxy"
Oct 30 19:09:23.044918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.044896    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.099170 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.099139    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"]
Oct 30 19:09:23.140614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.140567    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-cc079d-serving-cert\""
Oct 30 19:09:23.218862 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.218829    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.219047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.218966    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a1ddd5-52eb-4874-958d-432638ae01fb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.219102 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.219068    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-metrics-certs\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.219153 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.219140    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpql8\" (UniqueName: \"kubernetes.io/projected/b0a1ddd5-52eb-4874-958d-432638ae01fb-kube-api-access-tpql8\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.325759 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.324525    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpql8\" (UniqueName: \"kubernetes.io/projected/b0a1ddd5-52eb-4874-958d-432638ae01fb-kube-api-access-tpql8\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.325759 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.324705    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.325759 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.324829    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a1ddd5-52eb-4874-958d-432638ae01fb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.325759 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.324906    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-metrics-certs\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.338605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.335673    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a1ddd5-52eb-4874-958d-432638ae01fb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.348016 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.345007    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-metrics-certs\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.374271 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.368510    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.392226 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.392189    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpql8\" (UniqueName: \"kubernetes.io/projected/b0a1ddd5-52eb-4874-958d-432638ae01fb-kube-api-access-tpql8\") pod \"maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:23.660515 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:23.660490    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:24.042721 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:09:24.041852    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a1ddd5_52eb_4874_958d_432638ae01fb.slice/crio-cecef0b5532cb02d955da7b1768c99d4446f943c80a76397e82273cf42ece881 WatchSource:0}: Error finding container cecef0b5532cb02d955da7b1768c99d4446f943c80a76397e82273cf42ece881: Status 404 returned error can't find the container with id cecef0b5532cb02d955da7b1768c99d4446f943c80a76397e82273cf42ece881
Oct 30 19:09:24.045012 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:24.044982    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"]
Oct 30 19:09:24.357221 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:24.357149    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt" event={"ID":"b0a1ddd5-52eb-4874-958d-432638ae01fb","Type":"ContainerStarted","Data":"cecef0b5532cb02d955da7b1768c99d4446f943c80a76397e82273cf42ece881"}
Oct 30 19:09:25.409715 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:25.409677    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt" event={"ID":"b0a1ddd5-52eb-4874-958d-432638ae01fb","Type":"ContainerStarted","Data":"61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197"}
Oct 30 19:09:25.411401 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:25.411376    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:25.527368 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:25.527293    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt" podStartSLOduration=2.628720778 podStartE2EDuration="3.527272136s" totalImagesPullingTime="898.551358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:25.523098016 +0000 UTC m=+1960.633831300" watchObservedRunningTime="2026-10-30 19:09:25.527272136 +0000 UTC m=+1960.638005420"
Oct 30 19:09:32.484602 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:32.481965    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:09:32.869495 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:32.869410    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8873-e5ccb0"]
Oct 30 19:09:32.873989 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:32.873966    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8873-e5ccb0"
Oct 30 19:09:32.923746 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:32.923715    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qk8z9\""
Oct 30 19:09:32.924782 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:32.924748    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8873-e5ccb0"]
Oct 30 19:09:32.997713 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:32.997674    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vp6j\" (UniqueName: \"kubernetes.io/projected/ae3560fd-65cc-4736-ab4e-9d120430a217-kube-api-access-6vp6j\") pod \"test-curl-8873-e5ccb0\" (UID: \"ae3560fd-65cc-4736-ab4e-9d120430a217\") " pod="opendatahub/test-curl-8873-e5ccb0"
Oct 30 19:09:33.099273 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:33.099237    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vp6j\" (UniqueName: \"kubernetes.io/projected/ae3560fd-65cc-4736-ab4e-9d120430a217-kube-api-access-6vp6j\") pod \"test-curl-8873-e5ccb0\" (UID: \"ae3560fd-65cc-4736-ab4e-9d120430a217\") " pod="opendatahub/test-curl-8873-e5ccb0"
Oct 30 19:09:33.153568 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:33.153494    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vp6j\" (UniqueName: \"kubernetes.io/projected/ae3560fd-65cc-4736-ab4e-9d120430a217-kube-api-access-6vp6j\") pod \"test-curl-8873-e5ccb0\" (UID: \"ae3560fd-65cc-4736-ab4e-9d120430a217\") " pod="opendatahub/test-curl-8873-e5ccb0"
Oct 30 19:09:33.186872 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:33.186837    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8873-e5ccb0"
Oct 30 19:09:33.558522 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:33.558493    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8873-e5ccb0"]
Oct 30 19:09:33.559332 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:09:33.559300    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae3560fd_65cc_4736_ab4e_9d120430a217.slice/crio-84fc0c83467616b75cf5bdb3e060c015ea5ef87b93ff65906f0dc13123ad56d9 WatchSource:0}: Error finding container 84fc0c83467616b75cf5bdb3e060c015ea5ef87b93ff65906f0dc13123ad56d9: Status 404 returned error can't find the container with id 84fc0c83467616b75cf5bdb3e060c015ea5ef87b93ff65906f0dc13123ad56d9
Oct 30 19:09:34.492325 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:34.492286    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8873-e5ccb0" event={"ID":"ae3560fd-65cc-4736-ab4e-9d120430a217","Type":"ContainerStarted","Data":"ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a"}
Oct 30 19:09:34.492473 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:34.492336    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8873-e5ccb0" event={"ID":"ae3560fd-65cc-4736-ab4e-9d120430a217","Type":"ContainerStarted","Data":"84fc0c83467616b75cf5bdb3e060c015ea5ef87b93ff65906f0dc13123ad56d9"}
Oct 30 19:09:34.601539 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:34.601480    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8873-e5ccb0" podStartSLOduration=2.601465844 podStartE2EDuration="2.601465844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:34.598889795 +0000 UTC m=+1969.709623077" watchObservedRunningTime="2026-10-30 19:09:34.601465844 +0000 UTC m=+1969.712199127"
Oct 30 19:09:36.233296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:36.233260    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8873-e5ccb0"]
Oct 30 19:09:36.239396 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:36.239364    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8873-e5ccb0"]
Oct 30 19:09:36.506370 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:36.506267    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8873-e5ccb0" podUID="ae3560fd-65cc-4736-ab4e-9d120430a217" containerName="test-curl-8873-e5ccb0" containerID="cri-o://ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a" gracePeriod=2
Oct 30 19:09:36.554386 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:36.554344    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8873-e5ccb0\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="ae3560fd-65cc-4736-ab4e-9d120430a217" pod="opendatahub/test-curl-8873-e5ccb0"
Oct 30 19:09:38.745930 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:38.745894    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8873-e5ccb0"
Oct 30 19:09:38.774296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:38.774222    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3560fd-65cc-4736-ab4e-9d120430a217-kube-api-access-6vp6j\" (UniqueName: \"kubernetes.io/projected/ae3560fd-65cc-4736-ab4e-9d120430a217-kube-api-access-6vp6j\") pod \"ae3560fd-65cc-4736-ab4e-9d120430a217\" (UID: \"ae3560fd-65cc-4736-ab4e-9d120430a217\") "
Oct 30 19:09:38.776214 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:38.776186    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3560fd-65cc-4736-ab4e-9d120430a217-kube-api-access-6vp6j" pod "ae3560fd-65cc-4736-ab4e-9d120430a217" (UID: "ae3560fd-65cc-4736-ab4e-9d120430a217"). InnerVolumeSpecName "kube-api-access-6vp6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:38.887605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:38.881987    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vp6j\" (UniqueName: \"kubernetes.io/projected/ae3560fd-65cc-4736-ab4e-9d120430a217-kube-api-access-6vp6j\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:39.009246 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:39.008111    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3560fd-65cc-4736-ab4e-9d120430a217" path="/var/lib/kubelet/pods/ae3560fd-65cc-4736-ab4e-9d120430a217/volumes"
Oct 30 19:09:39.532857 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:39.532826    4391 generic.go:356] "Generic (PLEG): container finished" podID="ae3560fd-65cc-4736-ab4e-9d120430a217" containerID="ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a" exitCode=137
Oct 30 19:09:39.533063 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:39.532889    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8873-e5ccb0"
Oct 30 19:09:39.533063 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:39.532929    4391 scope.go:122] "RemoveContainer" containerID="ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a"
Oct 30 19:09:39.542966 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:39.542944    4391 scope.go:122] "RemoveContainer" containerID="ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a"
Oct 30 19:09:39.543262 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:39.543240    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a\": container with ID starting with ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a not found: ID does not exist" containerID="ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a"
Oct 30 19:09:39.543323 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:09:39.543272    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a"} err="failed to get container status \"ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a\": rpc error: code = NotFound desc = could not find container \"ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a\": container with ID starting with ace71c4b01c043b1bed66ffb9e4a003db786aa097fa0e022e3e1d1c399ed757a not found: ID does not exist"
Oct 30 19:09:41.255505 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:09:41.255470    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:10:00.907461 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:00.907426    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:10:05.346397 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.346362    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"]
Oct 30 19:10:05.347301 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.347278    4391 cpu_manager.go:414] "containerMap: removing container" podUID="ae3560fd-65cc-4736-ab4e-9d120430a217" containerName="test-curl-8873-e5ccb0"
Oct 30 19:10:05.347422 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.347301    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3560fd-65cc-4736-ab4e-9d120430a217" containerName="test-curl-8873-e5ccb0"
Oct 30 19:10:05.347511 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.347497    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3560fd-65cc-4736-ab4e-9d120430a217" containerName="test-curl-8873-e5ccb0"
Oct 30 19:10:05.353177 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.353153    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.384003 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.383970    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-metrics-certs\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.384187 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.384155    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-maas-api-tls\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.384280 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.384263    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gb7f\" (UniqueName: \"kubernetes.io/projected/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-kube-api-access-2gb7f\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.384317 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.384307    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-openshift-service-ca\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.419446 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.419407    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"]
Oct 30 19:10:05.447781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.447739    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-071cf0-serving-cert\""
Oct 30 19:10:05.485232 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.485189    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-maas-api-tls\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.485447 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.485290    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gb7f\" (UniqueName: \"kubernetes.io/projected/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-kube-api-access-2gb7f\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.485447 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.485330    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-openshift-service-ca\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.485447 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:10:05.485361    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-serving-cert: secret "maas-api-e2e-inf-a-071cf0-serving-cert" not found
Oct 30 19:10:05.485447 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:10:05.485441    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-maas-api-tls podName:ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757 nodeName:}" failed. No retries permitted until 2026-10-30 19:10:05.985418602 +0000 UTC m=+2001.096151880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-maas-api-tls") pod "maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww" (UID: "ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757") : secret "maas-api-e2e-inf-a-071cf0-serving-cert" not found
Oct 30 19:10:05.485705 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.485550    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-metrics-certs\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.486130 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.486107    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-openshift-service-ca\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.488229 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.488198    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-metrics-certs\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:05.539732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:05.539694    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gb7f\" (UniqueName: \"kubernetes.io/projected/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-kube-api-access-2gb7f\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:06.005758 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:06.005704    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-maas-api-tls\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:06.028604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:06.009464    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-maas-api-tls\") pod \"maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:06.270653 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:06.270531    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:06.661356 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:10:06.661325    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6e4d2a_a3c0_42e0_94c0_b797fa1cc757.slice/crio-9a76ee655064710b2183be82e1b0afa80f3a9bed038ded08241366fa22a56283 WatchSource:0}: Error finding container 9a76ee655064710b2183be82e1b0afa80f3a9bed038ded08241366fa22a56283: Status 404 returned error can't find the container with id 9a76ee655064710b2183be82e1b0afa80f3a9bed038ded08241366fa22a56283
Oct 30 19:10:06.664048 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:06.664020    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"]
Oct 30 19:10:06.838574 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:06.838512    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww" event={"ID":"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757","Type":"ContainerStarted","Data":"9a76ee655064710b2183be82e1b0afa80f3a9bed038ded08241366fa22a56283"}
Oct 30 19:10:07.847911 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:07.847809    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww" event={"ID":"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757","Type":"ContainerStarted","Data":"a62717e8833a3ef3a1e3a960a2a0db968b9fb096748abe1b9046c05d6b5d7dae"}
Oct 30 19:10:07.848330 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:07.847911    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:11.255165 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:10:11.255121    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:10:13.860391 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:13.860360    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:10:14.009603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:14.008822    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww" podStartSLOduration=8.121432652 podStartE2EDuration="9.008803921s" totalImagesPullingTime="887.371269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:07.962841568 +0000 UTC m=+2003.073574854" watchObservedRunningTime="2026-10-30 19:10:14.008803921 +0000 UTC m=+2009.119537207"
Oct 30 19:10:29.155046 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:29.154962    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"]
Oct 30 19:10:29.155527 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:29.155182    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy" containerID="cri-o://fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969" gracePeriod=30
Oct 30 19:10:33.613676 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:33.613637    4391 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:10:33.614152 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:33.613710    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:10:34.438207 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.438135    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 19:10:34.523251 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523214    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istiod-ca-cert\") pod \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") "
Oct 30 19:10:34.523430 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523401    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-certs\") pod \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") "
Oct 30 19:10:34.523490 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523441    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-socket\") pod \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") "
Oct 30 19:10:34.523557 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523498    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-envoy\") pod \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") "
Oct 30 19:10:34.523557 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523539    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-podinfo\") pod \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") "
Oct 30 19:10:34.523712 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523643    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-credential-socket\") pod \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") "
Oct 30 19:10:34.523772 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523707    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istiod-ca-cert" pod "93c04b3d-6c49-4b39-bf15-88b8c3a66540" (UID: "93c04b3d-6c49-4b39-bf15-88b8c3a66540"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:34.523772 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523767    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-kube-api-access-rgqjm\" (UniqueName: \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-kube-api-access-rgqjm\") pod \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") "
Oct 30 19:10:34.523888 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523841    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-token\" (UniqueName: \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-token\") pod \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") "
Oct 30 19:10:34.523888 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523846    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-certs" pod "93c04b3d-6c49-4b39-bf15-88b8c3a66540" (UID: "93c04b3d-6c49-4b39-bf15-88b8c3a66540"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.524007 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.523950    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-data\") pod \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\" (UID: \"93c04b3d-6c49-4b39-bf15-88b8c3a66540\") "
Oct 30 19:10:34.524728 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.524123    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-socket" pod "93c04b3d-6c49-4b39-bf15-88b8c3a66540" (UID: "93c04b3d-6c49-4b39-bf15-88b8c3a66540"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.524728 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.524297    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-credential-socket" pod "93c04b3d-6c49-4b39-bf15-88b8c3a66540" (UID: "93c04b3d-6c49-4b39-bf15-88b8c3a66540"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.524728 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.524361    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-data" pod "93c04b3d-6c49-4b39-bf15-88b8c3a66540" (UID: "93c04b3d-6c49-4b39-bf15-88b8c3a66540"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.525762 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.524743    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.525762 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.524765    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.525762 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.524779    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.525762 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.524792    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.525762 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.524804    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.526755 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.526734    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-envoy" pod "93c04b3d-6c49-4b39-bf15-88b8c3a66540" (UID: "93c04b3d-6c49-4b39-bf15-88b8c3a66540"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.526998 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.526980    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-kube-api-access-rgqjm" pod "93c04b3d-6c49-4b39-bf15-88b8c3a66540" (UID: "93c04b3d-6c49-4b39-bf15-88b8c3a66540"). InnerVolumeSpecName "kube-api-access-rgqjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:34.527702 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.527558    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-token" pod "93c04b3d-6c49-4b39-bf15-88b8c3a66540" (UID: "93c04b3d-6c49-4b39-bf15-88b8c3a66540"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:34.527702 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.527652    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-podinfo" pod "93c04b3d-6c49-4b39-bf15-88b8c3a66540" (UID: "93c04b3d-6c49-4b39-bf15-88b8c3a66540"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:10:34.626439 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.626396    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgqjm\" (UniqueName: \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-kube-api-access-rgqjm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.626439 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.626440    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.626880 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.626456    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.626880 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:34.626471    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c04b3d-6c49-4b39-bf15-88b8c3a66540-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:35.047801 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:35.047762    4391 generic.go:356] "Generic (PLEG): container finished" podID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerID="fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969" exitCode=0
Oct 30 19:10:35.047956 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:35.047820    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"
Oct 30 19:10:35.047956 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:35.047825    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" event={"ID":"93c04b3d-6c49-4b39-bf15-88b8c3a66540","Type":"ContainerDied","Data":"fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969"}
Oct 30 19:10:35.047956 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:35.047942    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb" event={"ID":"93c04b3d-6c49-4b39-bf15-88b8c3a66540","Type":"ContainerDied","Data":"c9afa1c8197da66031eb942bde31ebf4b16efc48148d52f3842102f290fb18a7"}
Oct 30 19:10:35.048084 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:35.047964    4391 scope.go:122] "RemoveContainer" containerID="fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969"
Oct 30 19:10:35.058306 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:35.058289    4391 scope.go:122] "RemoveContainer" containerID="fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969"
Oct 30 19:10:35.058632 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:10:35.058603    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969\": container with ID starting with fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969 not found: ID does not exist" containerID="fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969"
Oct 30 19:10:35.058702 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:35.058642    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969"} err="failed to get container status \"fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969\": rpc error: code = NotFound desc = could not find container \"fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969\": container with ID starting with fb185548726114433e36ee9f58f8991d066c551daa3917a53f7ba6d83395d969 not found: ID does not exist"
Oct 30 19:10:35.207641 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:35.207600    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"]
Oct 30 19:10:35.254361 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:35.254324    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-rj5rb"]
Oct 30 19:10:37.007876 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:37.007842    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" path="/var/lib/kubelet/pods/93c04b3d-6c49-4b39-bf15-88b8c3a66540/volumes"
Oct 30 19:10:41.253239 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:10:41.253201    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:10:54.811652 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:10:54.811613    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:11:11.254510 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:11:11.254472    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:11:12.253851 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.253797    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"]
Oct 30 19:11:12.254783 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.254752    4391 cpu_manager.go:414] "containerMap: removing container" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy"
Oct 30 19:11:12.254783 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.254776    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy"
Oct 30 19:11:12.255194 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.254883    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c04b3d-6c49-4b39-bf15-88b8c3a66540" containerName="istio-proxy"
Oct 30 19:11:12.258437 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.258416    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.313252 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.313213    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"]
Oct 30 19:11:12.350131 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.350085    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2057bc1a-888a-4706-83fb-1006e3954ada-openshift-service-ca\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.350367 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.350255    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rjm\" (UniqueName: \"kubernetes.io/projected/2057bc1a-888a-4706-83fb-1006e3954ada-kube-api-access-g5rjm\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.350367 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.350298    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-metrics-certs\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.350367 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.350359    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-maas-api-tls\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.353013 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.352985    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8e31a7-serving-cert\""
Oct 30 19:11:12.451979 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.451944    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2057bc1a-888a-4706-83fb-1006e3954ada-openshift-service-ca\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.452154 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.452097    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rjm\" (UniqueName: \"kubernetes.io/projected/2057bc1a-888a-4706-83fb-1006e3954ada-kube-api-access-g5rjm\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.452154 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.452135    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-metrics-certs\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.452270 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.452195    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-maas-api-tls\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.452477 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:11:12.452458    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-serving-cert: secret "maas-api-e2e-api-b-8e31a7-serving-cert" not found
Oct 30 19:11:12.452547 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:11:12.452541    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-maas-api-tls podName:2057bc1a-888a-4706-83fb-1006e3954ada nodeName:}" failed. No retries permitted until 2026-10-30 19:11:12.952521302 +0000 UTC m=+2068.063254566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-maas-api-tls") pod "maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2" (UID: "2057bc1a-888a-4706-83fb-1006e3954ada") : secret "maas-api-e2e-api-b-8e31a7-serving-cert" not found
Oct 30 19:11:12.452815 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.452784    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2057bc1a-888a-4706-83fb-1006e3954ada-openshift-service-ca\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.454855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.454829    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-metrics-certs\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:12.506930 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:12.506862    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rjm\" (UniqueName: \"kubernetes.io/projected/2057bc1a-888a-4706-83fb-1006e3954ada-kube-api-access-g5rjm\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:13.025861 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:13.025821    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-maas-api-tls\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:13.030915 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:13.030045    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-maas-api-tls\") pod \"maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:13.172990 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:13.172958    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:13.621629 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:13.621600    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"]
Oct 30 19:11:13.622625 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:11:13.622571    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2057bc1a_888a_4706_83fb_1006e3954ada.slice/crio-891e0908c0440f3d12928712c17babf678f051f872b8652ec1ad6d956fc4848f WatchSource:0}: Error finding container 891e0908c0440f3d12928712c17babf678f051f872b8652ec1ad6d956fc4848f: Status 404 returned error can't find the container with id 891e0908c0440f3d12928712c17babf678f051f872b8652ec1ad6d956fc4848f
Oct 30 19:11:13.624593 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:13.624557    4391 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:11:14.380842 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:14.380801    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2" event={"ID":"2057bc1a-888a-4706-83fb-1006e3954ada","Type":"ContainerStarted","Data":"891e0908c0440f3d12928712c17babf678f051f872b8652ec1ad6d956fc4848f"}
Oct 30 19:11:15.389956 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:15.389922    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2" event={"ID":"2057bc1a-888a-4706-83fb-1006e3954ada","Type":"ContainerStarted","Data":"f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f"}
Oct 30 19:11:15.390346 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:15.390020    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:15.500848 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:15.500789    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2" podStartSLOduration=2.615822244 podStartE2EDuration="3.500775588s" totalImagesPullingTime="884.953344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:15.498401464 +0000 UTC m=+2070.609134764" watchObservedRunningTime="2026-10-30 19:11:15.500775588 +0000 UTC m=+2070.611508872"
Oct 30 19:11:21.401714 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:21.401683    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:11:34.798276 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:34.798240    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:11:41.254786 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:11:41.254744    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:11:42.546716 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.546674    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"]
Oct 30 19:11:42.551089 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.551067    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.651614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.651565    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"]
Oct 30 19:11:42.651788 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.651749    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f3a0e7-serving-cert\""
Oct 30 19:11:42.716639 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.716596    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-metrics-certs\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.716854 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.716822    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpb7x\" (UniqueName: \"kubernetes.io/projected/c7ce2533-9241-4180-9675-fcbd391c3105-kube-api-access-lpb7x\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.716983 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.716960    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ce2533-9241-4180-9675-fcbd391c3105-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.717088 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.717070    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-maas-api-tls\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.827709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.827570    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-metrics-certs\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.828105 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.828075    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpb7x\" (UniqueName: \"kubernetes.io/projected/c7ce2533-9241-4180-9675-fcbd391c3105-kube-api-access-lpb7x\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.828371 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.828349    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ce2533-9241-4180-9675-fcbd391c3105-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.828800 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.828746    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-maas-api-tls\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.829219 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:11:42.829198    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-serving-cert: secret "maas-api-e2e-api-b-f3a0e7-serving-cert" not found
Oct 30 19:11:42.848609 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:11:42.829384    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-maas-api-tls podName:c7ce2533-9241-4180-9675-fcbd391c3105 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:43.329361543 +0000 UTC m=+2098.440094817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-maas-api-tls") pod "maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d" (UID: "c7ce2533-9241-4180-9675-fcbd391c3105") : secret "maas-api-e2e-api-b-f3a0e7-serving-cert" not found
Oct 30 19:11:42.848880 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.832265    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ce2533-9241-4180-9675-fcbd391c3105-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.864958 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.864925    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-metrics-certs\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:42.885217 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:42.885177    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpb7x\" (UniqueName: \"kubernetes.io/projected/c7ce2533-9241-4180-9675-fcbd391c3105-kube-api-access-lpb7x\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:43.337928 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:43.337875    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-maas-api-tls\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:43.340372 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:43.340343    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-maas-api-tls\") pod \"maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:43.466480 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:43.466438    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:43.842774 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:43.842745    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"]
Oct 30 19:11:43.843265 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:11:43.843241    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ce2533_9241_4180_9675_fcbd391c3105.slice/crio-8bde309278cedaed7fb9e0e420a80a2e785ec2f03cb38b50d93f042566f1f365 WatchSource:0}: Error finding container 8bde309278cedaed7fb9e0e420a80a2e785ec2f03cb38b50d93f042566f1f365: Status 404 returned error can't find the container with id 8bde309278cedaed7fb9e0e420a80a2e785ec2f03cb38b50d93f042566f1f365
Oct 30 19:11:44.629732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:44.629695    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d" event={"ID":"c7ce2533-9241-4180-9675-fcbd391c3105","Type":"ContainerStarted","Data":"8bde309278cedaed7fb9e0e420a80a2e785ec2f03cb38b50d93f042566f1f365"}
Oct 30 19:11:45.638032 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:45.637992    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d" event={"ID":"c7ce2533-9241-4180-9675-fcbd391c3105","Type":"ContainerStarted","Data":"ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc"}
Oct 30 19:11:45.638432 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:45.638081    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:11:45.749811 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:45.749757    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d" podStartSLOduration=2.905581639 podStartE2EDuration="3.749741575s" totalImagesPullingTime="844.159936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:45.745663582 +0000 UTC m=+2100.856396866" watchObservedRunningTime="2026-10-30 19:11:45.749741575 +0000 UTC m=+2100.860474857"
Oct 30 19:11:47.169454 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.169416    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"]
Oct 30 19:11:47.169964 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.169672    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7" podUID="568b301b-486f-4cf6-8d4a-26598ba2fef0" containerName="maas-api" containerID="cri-o://60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424" gracePeriod=30
Oct 30 19:11:47.591174 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.590807    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:11:47.699639 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.697845    4391 generic.go:356] "Generic (PLEG): container finished" podID="568b301b-486f-4cf6-8d4a-26598ba2fef0" containerID="60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424" exitCode=0
Oct 30 19:11:47.699639 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.697912    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7" event={"ID":"568b301b-486f-4cf6-8d4a-26598ba2fef0","Type":"ContainerDied","Data":"60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424"}
Oct 30 19:11:47.699639 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.697969    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7" event={"ID":"568b301b-486f-4cf6-8d4a-26598ba2fef0","Type":"ContainerDied","Data":"f4d508a7a3caf23c0ee2cc1fac5b101cc0ef98b85426211116e928995b709495"}
Oct 30 19:11:47.699639 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.698003    4391 scope.go:122] "RemoveContainer" containerID="60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424"
Oct 30 19:11:47.699639 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.698179    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"
Oct 30 19:11:47.761862 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.761820    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-maas-api-tls\") pod \"568b301b-486f-4cf6-8d4a-26598ba2fef0\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") "
Oct 30 19:11:47.762381 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.762357    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568b301b-486f-4cf6-8d4a-26598ba2fef0-kube-api-access-tll6v\" (UniqueName: \"kubernetes.io/projected/568b301b-486f-4cf6-8d4a-26598ba2fef0-kube-api-access-tll6v\") pod \"568b301b-486f-4cf6-8d4a-26598ba2fef0\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") "
Oct 30 19:11:47.762552 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.762537    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/568b301b-486f-4cf6-8d4a-26598ba2fef0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/568b301b-486f-4cf6-8d4a-26598ba2fef0-openshift-service-ca\") pod \"568b301b-486f-4cf6-8d4a-26598ba2fef0\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") "
Oct 30 19:11:47.762760 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.762741    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-metrics-certs\") pod \"568b301b-486f-4cf6-8d4a-26598ba2fef0\" (UID: \"568b301b-486f-4cf6-8d4a-26598ba2fef0\") "
Oct 30 19:11:47.769433 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.769058    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568b301b-486f-4cf6-8d4a-26598ba2fef0-openshift-service-ca" pod "568b301b-486f-4cf6-8d4a-26598ba2fef0" (UID: "568b301b-486f-4cf6-8d4a-26598ba2fef0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:47.774305 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.774277    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-metrics-certs" pod "568b301b-486f-4cf6-8d4a-26598ba2fef0" (UID: "568b301b-486f-4cf6-8d4a-26598ba2fef0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:47.775103 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.775074    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568b301b-486f-4cf6-8d4a-26598ba2fef0-kube-api-access-tll6v" pod "568b301b-486f-4cf6-8d4a-26598ba2fef0" (UID: "568b301b-486f-4cf6-8d4a-26598ba2fef0"). InnerVolumeSpecName "kube-api-access-tll6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:47.776071 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.776049    4391 scope.go:122] "RemoveContainer" containerID="60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424"
Oct 30 19:11:47.776529 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:11:47.776471    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424\": container with ID starting with 60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424 not found: ID does not exist" containerID="60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424"
Oct 30 19:11:47.776529 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.776511    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424"} err="failed to get container status \"60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424\": rpc error: code = NotFound desc = could not find container \"60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424\": container with ID starting with 60e63662ce72af2ef450185552e18e890aadf7638c305122aca60805e6f8b424 not found: ID does not exist"
Oct 30 19:11:47.776939 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.776922    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-maas-api-tls" pod "568b301b-486f-4cf6-8d4a-26598ba2fef0" (UID: "568b301b-486f-4cf6-8d4a-26598ba2fef0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:47.865335 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.865294    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:47.865335 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.865325    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tll6v\" (UniqueName: \"kubernetes.io/projected/568b301b-486f-4cf6-8d4a-26598ba2fef0-kube-api-access-tll6v\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:47.865335 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.865336    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/568b301b-486f-4cf6-8d4a-26598ba2fef0-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:47.865616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:47.865346    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/568b301b-486f-4cf6-8d4a-26598ba2fef0-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:48.157662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:48.156685    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"]
Oct 30 19:11:48.234603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:48.229108    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-wdht7"]
Oct 30 19:11:49.008212 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:49.008176    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568b301b-486f-4cf6-8d4a-26598ba2fef0" path="/var/lib/kubelet/pods/568b301b-486f-4cf6-8d4a-26598ba2fef0/volumes"
Oct 30 19:11:51.650599 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:11:51.650555    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:12:11.255010 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:12:11.254926    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:12:20.005510 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.005457    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"]
Oct 30 19:12:20.006100 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.005723    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j" podUID="50829478-c347-4223-96eb-63902b988f15" containerName="maas-api" containerID="cri-o://bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510" gracePeriod=30
Oct 30 19:12:20.464428 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.464380    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:12:20.513745 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.510741    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50829478-c347-4223-96eb-63902b988f15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50829478-c347-4223-96eb-63902b988f15-openshift-service-ca\") pod \"50829478-c347-4223-96eb-63902b988f15\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") "
Oct 30 19:12:20.513745 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.510973    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50829478-c347-4223-96eb-63902b988f15-kube-api-access-xkh2l\" (UniqueName: \"kubernetes.io/projected/50829478-c347-4223-96eb-63902b988f15-kube-api-access-xkh2l\") pod \"50829478-c347-4223-96eb-63902b988f15\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") "
Oct 30 19:12:20.513745 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.511036    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-maas-api-tls\") pod \"50829478-c347-4223-96eb-63902b988f15\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") "
Oct 30 19:12:20.513745 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.511131    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-metrics-certs\") pod \"50829478-c347-4223-96eb-63902b988f15\" (UID: \"50829478-c347-4223-96eb-63902b988f15\") "
Oct 30 19:12:20.513745 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.512657    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50829478-c347-4223-96eb-63902b988f15-openshift-service-ca" pod "50829478-c347-4223-96eb-63902b988f15" (UID: "50829478-c347-4223-96eb-63902b988f15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:20.548151 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.548024    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50829478-c347-4223-96eb-63902b988f15-kube-api-access-xkh2l" pod "50829478-c347-4223-96eb-63902b988f15" (UID: "50829478-c347-4223-96eb-63902b988f15"). InnerVolumeSpecName "kube-api-access-xkh2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:20.548151 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.548114    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-maas-api-tls" pod "50829478-c347-4223-96eb-63902b988f15" (UID: "50829478-c347-4223-96eb-63902b988f15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:20.551389 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.551025    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-metrics-certs" pod "50829478-c347-4223-96eb-63902b988f15" (UID: "50829478-c347-4223-96eb-63902b988f15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:20.613737 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.613656    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkh2l\" (UniqueName: \"kubernetes.io/projected/50829478-c347-4223-96eb-63902b988f15-kube-api-access-xkh2l\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:20.613737 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.613695    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:20.613737 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.613711    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50829478-c347-4223-96eb-63902b988f15-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:20.613737 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:20.613725    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50829478-c347-4223-96eb-63902b988f15-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.103605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:21.099400    4391 generic.go:356] "Generic (PLEG): container finished" podID="50829478-c347-4223-96eb-63902b988f15" containerID="bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510" exitCode=0
Oct 30 19:12:21.103605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:21.099704    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j" event={"ID":"50829478-c347-4223-96eb-63902b988f15","Type":"ContainerDied","Data":"bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510"}
Oct 30 19:12:21.103605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:21.099780    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j" event={"ID":"50829478-c347-4223-96eb-63902b988f15","Type":"ContainerDied","Data":"37a3ef628b0e71b989f7033a01155656141b7974ee28093be6461155d64838ef"}
Oct 30 19:12:21.103605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:21.099810    4391 scope.go:122] "RemoveContainer" containerID="bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510"
Oct 30 19:12:21.103605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:21.100049    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"
Oct 30 19:12:21.150318 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:21.149965    4391 scope.go:122] "RemoveContainer" containerID="bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510"
Oct 30 19:12:21.150705 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:12:21.150589    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510\": container with ID starting with bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510 not found: ID does not exist" containerID="bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510"
Oct 30 19:12:21.150705 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:21.150634    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510"} err="failed to get container status \"bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510\": rpc error: code = NotFound desc = could not find container \"bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510\": container with ID starting with bc1c5e7340d04e19a167e71c63af37364df8a61557906f4753534640e1143510 not found: ID does not exist"
Oct 30 19:12:21.289313 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:21.289273    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"]
Oct 30 19:12:21.355659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:21.352556    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cfb433-657b9469-zl59j"]
Oct 30 19:12:23.008269 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:23.008230    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50829478-c347-4223-96eb-63902b988f15" path="/var/lib/kubelet/pods/50829478-c347-4223-96eb-63902b988f15/volumes"
Oct 30 19:12:26.452667 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:26.452627    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"]
Oct 30 19:12:26.453116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:26.452901    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt" podUID="b0a1ddd5-52eb-4874-958d-432638ae01fb" containerName="maas-api" containerID="cri-o://61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197" gracePeriod=30
Oct 30 19:12:26.885451 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:26.885416    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:12:27.023598 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.023537    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a1ddd5-52eb-4874-958d-432638ae01fb-kube-api-access-tpql8\" (UniqueName: \"kubernetes.io/projected/b0a1ddd5-52eb-4874-958d-432638ae01fb-kube-api-access-tpql8\") pod \"b0a1ddd5-52eb-4874-958d-432638ae01fb\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") "
Oct 30 19:12:27.040760 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.040726    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-metrics-certs\") pod \"b0a1ddd5-52eb-4874-958d-432638ae01fb\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") "
Oct 30 19:12:27.042139 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.042113    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-maas-api-tls\") pod \"b0a1ddd5-52eb-4874-958d-432638ae01fb\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") "
Oct 30 19:12:27.042352 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.042336    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a1ddd5-52eb-4874-958d-432638ae01fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a1ddd5-52eb-4874-958d-432638ae01fb-openshift-service-ca\") pod \"b0a1ddd5-52eb-4874-958d-432638ae01fb\" (UID: \"b0a1ddd5-52eb-4874-958d-432638ae01fb\") "
Oct 30 19:12:27.043526 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.043502    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a1ddd5-52eb-4874-958d-432638ae01fb-openshift-service-ca" pod "b0a1ddd5-52eb-4874-958d-432638ae01fb" (UID: "b0a1ddd5-52eb-4874-958d-432638ae01fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:27.094854 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.094796    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-metrics-certs" pod "b0a1ddd5-52eb-4874-958d-432638ae01fb" (UID: "b0a1ddd5-52eb-4874-958d-432638ae01fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:27.097824 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.097782    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a1ddd5-52eb-4874-958d-432638ae01fb-kube-api-access-tpql8" pod "b0a1ddd5-52eb-4874-958d-432638ae01fb" (UID: "b0a1ddd5-52eb-4874-958d-432638ae01fb"). InnerVolumeSpecName "kube-api-access-tpql8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:27.107997 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.107903    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-maas-api-tls" pod "b0a1ddd5-52eb-4874-958d-432638ae01fb" (UID: "b0a1ddd5-52eb-4874-958d-432638ae01fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:27.152721 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.152691    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:27.152950 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.152922    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a1ddd5-52eb-4874-958d-432638ae01fb-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:27.153052 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.152955    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpql8\" (UniqueName: \"kubernetes.io/projected/b0a1ddd5-52eb-4874-958d-432638ae01fb-kube-api-access-tpql8\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:27.153052 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.152973    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0a1ddd5-52eb-4874-958d-432638ae01fb-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:27.153552 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.153525    4391 generic.go:356] "Generic (PLEG): container finished" podID="b0a1ddd5-52eb-4874-958d-432638ae01fb" containerID="61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197" exitCode=0
Oct 30 19:12:27.153688 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.153596    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"
Oct 30 19:12:27.153688 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.153605    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt" event={"ID":"b0a1ddd5-52eb-4874-958d-432638ae01fb","Type":"ContainerDied","Data":"61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197"}
Oct 30 19:12:27.153688 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.153654    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt" event={"ID":"b0a1ddd5-52eb-4874-958d-432638ae01fb","Type":"ContainerDied","Data":"cecef0b5532cb02d955da7b1768c99d4446f943c80a76397e82273cf42ece881"}
Oct 30 19:12:27.153688 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.153675    4391 scope.go:122] "RemoveContainer" containerID="61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197"
Oct 30 19:12:27.164874 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.164849    4391 scope.go:122] "RemoveContainer" containerID="61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197"
Oct 30 19:12:27.165160 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:12:27.165127    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197\": container with ID starting with 61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197 not found: ID does not exist" containerID="61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197"
Oct 30 19:12:27.165235 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.165164    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197"} err="failed to get container status \"61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197\": rpc error: code = NotFound desc = could not find container \"61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197\": container with ID starting with 61f5012dd21411a561e985833a099ccc1531de9c21c33fcb7bd970a970bb5197 not found: ID does not exist"
Oct 30 19:12:27.312408 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.312374    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"]
Oct 30 19:12:27.318895 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:27.318867    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cc079d-74f566bc79-q7fzt"]
Oct 30 19:12:29.007654 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:29.007621    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a1ddd5-52eb-4874-958d-432638ae01fb" path="/var/lib/kubelet/pods/b0a1ddd5-52eb-4874-958d-432638ae01fb/volumes"
Oct 30 19:12:31.695690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:31.695658    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"]
Oct 30 19:12:31.696118 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:31.695927    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy" containerID="cri-o://912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214" gracePeriod=30
Oct 30 19:12:34.695329 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:34.695287    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:12:34.696071 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:34.696033    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:12:37.029366 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.029339    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:12:37.155803 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.155769    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istiod-ca-cert\") pod \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") "
Oct 30 19:12:37.155957 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.155816    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-certs\") pod \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") "
Oct 30 19:12:37.155957 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.155855    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-token\" (UniqueName: \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-token\") pod \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") "
Oct 30 19:12:37.155957 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.155938    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-data\") pod \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") "
Oct 30 19:12:37.156116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.155970    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-envoy\") pod \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") "
Oct 30 19:12:37.156116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.156003    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-credential-socket\") pod \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") "
Oct 30 19:12:37.156224 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.156117    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-podinfo\") pod \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") "
Oct 30 19:12:37.156279 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.156227    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-kube-api-access-nwckz\" (UniqueName: \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-kube-api-access-nwckz\") pod \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") "
Oct 30 19:12:37.156279 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.156232    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-certs" pod "1861d4e6-403e-4ebb-a619-87bc3bb7efb7" (UID: "1861d4e6-403e-4ebb-a619-87bc3bb7efb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:37.156279 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.156266    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istiod-ca-cert" pod "1861d4e6-403e-4ebb-a619-87bc3bb7efb7" (UID: "1861d4e6-403e-4ebb-a619-87bc3bb7efb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:37.156438 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.156270    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-data" pod "1861d4e6-403e-4ebb-a619-87bc3bb7efb7" (UID: "1861d4e6-403e-4ebb-a619-87bc3bb7efb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:37.156438 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.156343    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-credential-socket" pod "1861d4e6-403e-4ebb-a619-87bc3bb7efb7" (UID: "1861d4e6-403e-4ebb-a619-87bc3bb7efb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:37.156550 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.156531    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-socket\") pod \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\" (UID: \"1861d4e6-403e-4ebb-a619-87bc3bb7efb7\") "
Oct 30 19:12:37.156789 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.156769    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-socket" pod "1861d4e6-403e-4ebb-a619-87bc3bb7efb7" (UID: "1861d4e6-403e-4ebb-a619-87bc3bb7efb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:37.157483 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.157464    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:37.157614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.157487    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:37.157614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.157503    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:37.157614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.157516    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:37.157614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.157529    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:37.158400 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.158374    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-podinfo" pod "1861d4e6-403e-4ebb-a619-87bc3bb7efb7" (UID: "1861d4e6-403e-4ebb-a619-87bc3bb7efb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:12:37.158499 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.158435    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-envoy" pod "1861d4e6-403e-4ebb-a619-87bc3bb7efb7" (UID: "1861d4e6-403e-4ebb-a619-87bc3bb7efb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:37.158658 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.158641    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-kube-api-access-nwckz" pod "1861d4e6-403e-4ebb-a619-87bc3bb7efb7" (UID: "1861d4e6-403e-4ebb-a619-87bc3bb7efb7"). InnerVolumeSpecName "kube-api-access-nwckz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:37.158708 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.158656    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-token" pod "1861d4e6-403e-4ebb-a619-87bc3bb7efb7" (UID: "1861d4e6-403e-4ebb-a619-87bc3bb7efb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:37.231490 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.231449    4391 generic.go:356] "Generic (PLEG): container finished" podID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerID="912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214" exitCode=0
Oct 30 19:12:37.231490 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.231493    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"
Oct 30 19:12:37.231780 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.231530    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" event={"ID":"1861d4e6-403e-4ebb-a619-87bc3bb7efb7","Type":"ContainerDied","Data":"912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214"}
Oct 30 19:12:37.231780 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.231643    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds" event={"ID":"1861d4e6-403e-4ebb-a619-87bc3bb7efb7","Type":"ContainerDied","Data":"4a3b8f7c8e51a8cd64f1f8a73c9f97df870ab7a674ee5248617c8fe5aae76cf7"}
Oct 30 19:12:37.231780 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.231662    4391 scope.go:122] "RemoveContainer" containerID="912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214"
Oct 30 19:12:37.248620 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.248593    4391 scope.go:122] "RemoveContainer" containerID="912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214"
Oct 30 19:12:37.248923 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:12:37.248903    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214\": container with ID starting with 912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214 not found: ID does not exist" containerID="912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214"
Oct 30 19:12:37.249016 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.248934    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214"} err="failed to get container status \"912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214\": rpc error: code = NotFound desc = could not find container \"912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214\": container with ID starting with 912696e83c84eec1e13b38e0abae5a1679d42d5e307abc6d315c996517514214 not found: ID does not exist"
Oct 30 19:12:37.258635 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.258607    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:37.258635 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.258636    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:37.258829 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.258649    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:37.258829 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.258665    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwckz\" (UniqueName: \"kubernetes.io/projected/1861d4e6-403e-4ebb-a619-87bc3bb7efb7-kube-api-access-nwckz\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:37.391098 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.391061    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"]
Oct 30 19:12:37.397912 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:37.397869    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-4mlds"]
Oct 30 19:12:39.012400 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:39.012362    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" path="/var/lib/kubelet/pods/1861d4e6-403e-4ebb-a619-87bc3bb7efb7/volumes"
Oct 30 19:12:41.315219 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:12:41.315177    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:12:49.007945 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.007912    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:12:49.838916 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.838881    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"]
Oct 30 19:12:49.839528 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839513    4391 cpu_manager.go:414] "containerMap: removing container" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy"
Oct 30 19:12:49.839614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839529    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy"
Oct 30 19:12:49.839614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839604    4391 cpu_manager.go:414] "containerMap: removing container" podUID="b0a1ddd5-52eb-4874-958d-432638ae01fb" containerName="maas-api"
Oct 30 19:12:49.839748 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839614    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a1ddd5-52eb-4874-958d-432638ae01fb" containerName="maas-api"
Oct 30 19:12:49.839748 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839640    4391 cpu_manager.go:414] "containerMap: removing container" podUID="50829478-c347-4223-96eb-63902b988f15" containerName="maas-api"
Oct 30 19:12:49.839748 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839649    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50829478-c347-4223-96eb-63902b988f15" containerName="maas-api"
Oct 30 19:12:49.839748 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839676    4391 cpu_manager.go:414] "containerMap: removing container" podUID="568b301b-486f-4cf6-8d4a-26598ba2fef0" containerName="maas-api"
Oct 30 19:12:49.839748 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839684    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568b301b-486f-4cf6-8d4a-26598ba2fef0" containerName="maas-api"
Oct 30 19:12:49.839940 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839857    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a1ddd5-52eb-4874-958d-432638ae01fb" containerName="maas-api"
Oct 30 19:12:49.839940 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839877    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="50829478-c347-4223-96eb-63902b988f15" containerName="maas-api"
Oct 30 19:12:49.839940 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839890    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="568b301b-486f-4cf6-8d4a-26598ba2fef0" containerName="maas-api"
Oct 30 19:12:49.839940 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.839903    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="1861d4e6-403e-4ebb-a619-87bc3bb7efb7" containerName="istio-proxy"
Oct 30 19:12:49.844786 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.844767    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:49.941033 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.941003    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-568107-serving-cert\""
Oct 30 19:12:49.944983 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.944956    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"]
Oct 30 19:12:49.965590 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.965558    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3537b0-91de-458c-b368-50e83eb38ee1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:49.965777 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.965755    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-maas-api-tls\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:49.965871 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.965855    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmkt\" (UniqueName: \"kubernetes.io/projected/2b3537b0-91de-458c-b368-50e83eb38ee1-kube-api-access-bjmkt\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:49.965991 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:49.965977    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-metrics-certs\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:50.067762 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:50.067728    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-maas-api-tls\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:50.068196 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:50.067821    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmkt\" (UniqueName: \"kubernetes.io/projected/2b3537b0-91de-458c-b368-50e83eb38ee1-kube-api-access-bjmkt\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:50.068196 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:12:50.067879    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-serving-cert: secret "maas-api-e2e-inf-a-568107-serving-cert" not found
Oct 30 19:12:50.068196 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:12:50.067941    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-maas-api-tls podName:2b3537b0-91de-458c-b368-50e83eb38ee1 nodeName:}" failed. No retries permitted until 2026-10-30 19:12:50.567921899 +0000 UTC m=+2165.678655176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-maas-api-tls") pod "maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh" (UID: "2b3537b0-91de-458c-b368-50e83eb38ee1") : secret "maas-api-e2e-inf-a-568107-serving-cert" not found
Oct 30 19:12:50.068196 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:50.067986    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-metrics-certs\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:50.068196 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:50.068119    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3537b0-91de-458c-b368-50e83eb38ee1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:50.068868 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:50.068846    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3537b0-91de-458c-b368-50e83eb38ee1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:50.070450 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:50.070430    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-metrics-certs\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:50.148738 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:50.142410    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmkt\" (UniqueName: \"kubernetes.io/projected/2b3537b0-91de-458c-b368-50e83eb38ee1-kube-api-access-bjmkt\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:50.588393 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:50.588359    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-maas-api-tls\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:50.590765 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:50.590739    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-maas-api-tls\") pod \"maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:50.759202 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:50.759143    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:51.141746 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:51.141709    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"]
Oct 30 19:12:51.142838 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:12:51.142815    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3537b0_91de_458c_b368_50e83eb38ee1.slice/crio-d7864bebd23244c99d46125dd9b4e264d2aaa5964547b606a36206f82ee19d63 WatchSource:0}: Error finding container d7864bebd23244c99d46125dd9b4e264d2aaa5964547b606a36206f82ee19d63: Status 404 returned error can't find the container with id d7864bebd23244c99d46125dd9b4e264d2aaa5964547b606a36206f82ee19d63
Oct 30 19:12:51.383847 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:51.383806    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh" event={"ID":"2b3537b0-91de-458c-b368-50e83eb38ee1","Type":"ContainerStarted","Data":"d7864bebd23244c99d46125dd9b4e264d2aaa5964547b606a36206f82ee19d63"}
Oct 30 19:12:52.393475 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:52.393367    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh" event={"ID":"2b3537b0-91de-458c-b368-50e83eb38ee1","Type":"ContainerStarted","Data":"53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2"}
Oct 30 19:12:52.393475 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:52.393450    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:12:52.505086 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:52.505030    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh" podStartSLOduration=2.642935687 podStartE2EDuration="3.505014642s" totalImagesPullingTime="862.078955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:52.50413024 +0000 UTC m=+2167.614863523" watchObservedRunningTime="2026-10-30 19:12:52.505014642 +0000 UTC m=+2167.615747924"
Oct 30 19:12:58.404951 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:12:58.404923    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:13:11.255034 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:13:11.254995    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:13:15.314727 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.314686    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"]
Oct 30 19:13:15.315278 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.314945    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww" podUID="ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757" containerName="maas-api" containerID="cri-o://a62717e8833a3ef3a1e3a960a2a0db968b9fb096748abe1b9046c05d6b5d7dae" gracePeriod=30
Oct 30 19:13:15.636242 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.630351    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"]
Oct 30 19:13:15.636242 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.630618    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy" containerID="cri-o://2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483" gracePeriod=30
Oct 30 19:13:15.684189 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.681950    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"]
Oct 30 19:13:15.684189 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.682178    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy" containerID="cri-o://173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb" gracePeriod=30
Oct 30 19:13:15.688985 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.688780    4391 generic.go:356] "Generic (PLEG): container finished" podID="ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757" containerID="a62717e8833a3ef3a1e3a960a2a0db968b9fb096748abe1b9046c05d6b5d7dae" exitCode=0
Oct 30 19:13:15.688985 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.688862    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww" event={"ID":"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757","Type":"ContainerDied","Data":"a62717e8833a3ef3a1e3a960a2a0db968b9fb096748abe1b9046c05d6b5d7dae"}
Oct 30 19:13:15.689874 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.689847    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"]
Oct 30 19:13:15.690250 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.690188    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy" containerID="cri-o://7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7" gracePeriod=30
Oct 30 19:13:15.742341 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.742305    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:13:15.782995 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.782953    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-maas-api-tls\") pod \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") "
Oct 30 19:13:15.783201 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.783044    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-kube-api-access-2gb7f\" (UniqueName: \"kubernetes.io/projected/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-kube-api-access-2gb7f\") pod \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") "
Oct 30 19:13:15.783278 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.783259    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-metrics-certs\") pod \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") "
Oct 30 19:13:15.783343 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.783328    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-openshift-service-ca\") pod \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\" (UID: \"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757\") "
Oct 30 19:13:15.784400 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.784369    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-openshift-service-ca" pod "ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757" (UID: "ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:15.786155 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.786132    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-maas-api-tls" pod "ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757" (UID: "ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:15.786456 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.786424    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-kube-api-access-2gb7f" pod "ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757" (UID: "ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757"). InnerVolumeSpecName "kube-api-access-2gb7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:15.788178 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.788155    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-metrics-certs" pod "ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757" (UID: "ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:15.885101 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.885067    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gb7f\" (UniqueName: \"kubernetes.io/projected/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-kube-api-access-2gb7f\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.885101 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.885099    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.885101 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.885109    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.885350 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:15.885120    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.696927 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:16.696891    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"
Oct 30 19:13:16.696927 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:16.696905    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww" event={"ID":"ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757","Type":"ContainerDied","Data":"9a76ee655064710b2183be82e1b0afa80f3a9bed038ded08241366fa22a56283"}
Oct 30 19:13:16.697432 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:16.696976    4391 scope.go:122] "RemoveContainer" containerID="a62717e8833a3ef3a1e3a960a2a0db968b9fb096748abe1b9046c05d6b5d7dae"
Oct 30 19:13:16.856372 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:16.856340    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"]
Oct 30 19:13:16.862918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:16.862886    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-071cf0-54d5ddd7bb-9lfww"]
Oct 30 19:13:17.007170 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:17.007075    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757" path="/var/lib/kubelet/pods/ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757/volumes"
Oct 30 19:13:18.268947 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:18.268909    4391 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:18.269401 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:18.268966    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:18.588492 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:18.588404    4391 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:18.588492 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:18.588473    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:18.592282 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:18.592252    4391 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:18.592381 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:18.592294    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:21.005612 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.005571    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 19:13:21.039358 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.039334    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 19:13:21.043273 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.043251    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.081945    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-socket\") pod \"eb90e7fe-ff29-4428-a376-4b5766eac25a\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082026    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-data\") pod \"ea6f7fdb-32a8-479b-a026-224b18d9532f\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082063    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6f7fdb-32a8-479b-a026-224b18d9532f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6f7fdb-32a8-479b-a026-224b18d9532f-istiod-ca-cert\") pod \"ea6f7fdb-32a8-479b-a026-224b18d9532f\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082150    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-podinfo\") pod \"eb90e7fe-ff29-4428-a376-4b5766eac25a\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082190    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-certs\") pod \"ea6f7fdb-32a8-479b-a026-224b18d9532f\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082288    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-podinfo\") pod \"ea6f7fdb-32a8-479b-a026-224b18d9532f\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082328    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-token\" (UniqueName: \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-token\") pod \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082377    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-envoy\") pod \"ea6f7fdb-32a8-479b-a026-224b18d9532f\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082421    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-data\") pod \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082528    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-socket\") pod \"ea6f7fdb-32a8-479b-a026-224b18d9532f\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082568    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istiod-ca-cert\") pod \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082631    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-certs\") pod \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082682    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-socket\") pod \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082732    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-data\") pod \"eb90e7fe-ff29-4428-a376-4b5766eac25a\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082733    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-socket" pod "eb90e7fe-ff29-4428-a376-4b5766eac25a" (UID: "eb90e7fe-ff29-4428-a376-4b5766eac25a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082764    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-certs\") pod \"eb90e7fe-ff29-4428-a376-4b5766eac25a\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082808    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-credential-socket\") pod \"eb90e7fe-ff29-4428-a376-4b5766eac25a\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082846    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-kube-api-access-v82w5\" (UniqueName: \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-kube-api-access-v82w5\") pod \"ea6f7fdb-32a8-479b-a026-224b18d9532f\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082895    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-token\" (UniqueName: \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-token\") pod \"ea6f7fdb-32a8-479b-a026-224b18d9532f\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082942    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-credential-socket\") pod \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.082982    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-kube-api-access-7sv59\" (UniqueName: \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-kube-api-access-7sv59\") pod \"eb90e7fe-ff29-4428-a376-4b5766eac25a\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.083033    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-token\" (UniqueName: \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-token\") pod \"eb90e7fe-ff29-4428-a376-4b5766eac25a\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.083077    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb90e7fe-ff29-4428-a376-4b5766eac25a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb90e7fe-ff29-4428-a376-4b5766eac25a-istiod-ca-cert\") pod \"eb90e7fe-ff29-4428-a376-4b5766eac25a\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.083118    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-envoy\") pod \"eb90e7fe-ff29-4428-a376-4b5766eac25a\" (UID: \"eb90e7fe-ff29-4428-a376-4b5766eac25a\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.083186    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-envoy\") pod \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.083249    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-podinfo\") pod \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.083277    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-credential-socket\") pod \"ea6f7fdb-32a8-479b-a026-224b18d9532f\" (UID: \"ea6f7fdb-32a8-479b-a026-224b18d9532f\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.083338    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-kube-api-access-24k5j\" (UniqueName: \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-kube-api-access-24k5j\") pod \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\" (UID: \"e20aeee3-1a1d-43cc-879e-4f0bac0e7601\") "
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.084303    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.088112    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-podinfo" pod "eb90e7fe-ff29-4428-a376-4b5766eac25a" (UID: "eb90e7fe-ff29-4428-a376-4b5766eac25a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.088526    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-envoy" pod "ea6f7fdb-32a8-479b-a026-224b18d9532f" (UID: "ea6f7fdb-32a8-479b-a026-224b18d9532f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.092782    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6f7fdb-32a8-479b-a026-224b18d9532f-istiod-ca-cert" pod "ea6f7fdb-32a8-479b-a026-224b18d9532f" (UID: "ea6f7fdb-32a8-479b-a026-224b18d9532f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.094783    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-token" pod "e20aeee3-1a1d-43cc-879e-4f0bac0e7601" (UID: "e20aeee3-1a1d-43cc-879e-4f0bac0e7601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.097857    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-kube-api-access-24k5j" pod "e20aeee3-1a1d-43cc-879e-4f0bac0e7601" (UID: "e20aeee3-1a1d-43cc-879e-4f0bac0e7601"). InnerVolumeSpecName "kube-api-access-24k5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.084901    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-certs" pod "ea6f7fdb-32a8-479b-a026-224b18d9532f" (UID: "ea6f7fdb-32a8-479b-a026-224b18d9532f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.099399    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-data" pod "ea6f7fdb-32a8-479b-a026-224b18d9532f" (UID: "ea6f7fdb-32a8-479b-a026-224b18d9532f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.100505    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-certs" pod "e20aeee3-1a1d-43cc-879e-4f0bac0e7601" (UID: "e20aeee3-1a1d-43cc-879e-4f0bac0e7601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.101302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.100639    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-data" pod "e20aeee3-1a1d-43cc-879e-4f0bac0e7601" (UID: "e20aeee3-1a1d-43cc-879e-4f0bac0e7601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.109766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.101936    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-socket" pod "ea6f7fdb-32a8-479b-a026-224b18d9532f" (UID: "ea6f7fdb-32a8-479b-a026-224b18d9532f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.109766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.102081    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istiod-ca-cert" pod "e20aeee3-1a1d-43cc-879e-4f0bac0e7601" (UID: "e20aeee3-1a1d-43cc-879e-4f0bac0e7601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:21.109766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.102366    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-socket" pod "e20aeee3-1a1d-43cc-879e-4f0bac0e7601" (UID: "e20aeee3-1a1d-43cc-879e-4f0bac0e7601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.109766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.102763    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-credential-socket" pod "ea6f7fdb-32a8-479b-a026-224b18d9532f" (UID: "ea6f7fdb-32a8-479b-a026-224b18d9532f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.109766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.104391    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-credential-socket" pod "e20aeee3-1a1d-43cc-879e-4f0bac0e7601" (UID: "e20aeee3-1a1d-43cc-879e-4f0bac0e7601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.111223 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.111043    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb90e7fe-ff29-4428-a376-4b5766eac25a-istiod-ca-cert" pod "eb90e7fe-ff29-4428-a376-4b5766eac25a" (UID: "eb90e7fe-ff29-4428-a376-4b5766eac25a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:21.111223 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.111111    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-certs" pod "eb90e7fe-ff29-4428-a376-4b5766eac25a" (UID: "eb90e7fe-ff29-4428-a376-4b5766eac25a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.111223 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.111158    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-credential-socket" pod "eb90e7fe-ff29-4428-a376-4b5766eac25a" (UID: "eb90e7fe-ff29-4428-a376-4b5766eac25a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.111223 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.111203    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-data" pod "eb90e7fe-ff29-4428-a376-4b5766eac25a" (UID: "eb90e7fe-ff29-4428-a376-4b5766eac25a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188244    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188280    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188295    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188308    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188325    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188338    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188351    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb90e7fe-ff29-4428-a376-4b5766eac25a-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188365    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188380    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24k5j\" (UniqueName: \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-kube-api-access-24k5j\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188395    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188410    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6f7fdb-32a8-479b-a026-224b18d9532f-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188426    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188438    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188452    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188468    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188481    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188495    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6f7fdb-32a8-479b-a026-224b18d9532f-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.188508    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.200047 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.193008    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-podinfo" pod "ea6f7fdb-32a8-479b-a026-224b18d9532f" (UID: "ea6f7fdb-32a8-479b-a026-224b18d9532f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:21.201429 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.201392    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-kube-api-access-7sv59" pod "eb90e7fe-ff29-4428-a376-4b5766eac25a" (UID: "eb90e7fe-ff29-4428-a376-4b5766eac25a"). InnerVolumeSpecName "kube-api-access-7sv59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.203662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.203466    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-envoy" pod "e20aeee3-1a1d-43cc-879e-4f0bac0e7601" (UID: "e20aeee3-1a1d-43cc-879e-4f0bac0e7601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.203662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.203545    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-podinfo" pod "e20aeee3-1a1d-43cc-879e-4f0bac0e7601" (UID: "e20aeee3-1a1d-43cc-879e-4f0bac0e7601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:21.203662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.203622    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-envoy" pod "eb90e7fe-ff29-4428-a376-4b5766eac25a" (UID: "eb90e7fe-ff29-4428-a376-4b5766eac25a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.203662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.203653    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-kube-api-access-v82w5" pod "ea6f7fdb-32a8-479b-a026-224b18d9532f" (UID: "ea6f7fdb-32a8-479b-a026-224b18d9532f"). InnerVolumeSpecName "kube-api-access-v82w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.203946 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.203810    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-token" pod "ea6f7fdb-32a8-479b-a026-224b18d9532f" (UID: "ea6f7fdb-32a8-479b-a026-224b18d9532f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.230833 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.230792    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-token" pod "eb90e7fe-ff29-4428-a376-4b5766eac25a" (UID: "eb90e7fe-ff29-4428-a376-4b5766eac25a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.290115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.290079    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v82w5\" (UniqueName: \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-kube-api-access-v82w5\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.290312 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.290151    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.290312 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.290171    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sv59\" (UniqueName: \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-kube-api-access-7sv59\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.290312 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.290186    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.290312 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.290200    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb90e7fe-ff29-4428-a376-4b5766eac25a-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.290312 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.290214    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.290312 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.290227    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20aeee3-1a1d-43cc-879e-4f0bac0e7601-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.290312 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.290239    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6f7fdb-32a8-479b-a026-224b18d9532f-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.737197 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.737161    4391 generic.go:356] "Generic (PLEG): container finished" podID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerID="173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb" exitCode=0
Oct 30 19:13:21.737388 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.737241    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"
Oct 30 19:13:21.737388 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.737247    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" event={"ID":"ea6f7fdb-32a8-479b-a026-224b18d9532f","Type":"ContainerDied","Data":"173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb"}
Oct 30 19:13:21.737388 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.737321    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt" event={"ID":"ea6f7fdb-32a8-479b-a026-224b18d9532f","Type":"ContainerDied","Data":"8c0fb79edcb88a79a93be3856ef7850c9612d28f0f115026bed465ee87196005"}
Oct 30 19:13:21.737388 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.737351    4391 scope.go:122] "RemoveContainer" containerID="173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb"
Oct 30 19:13:21.739120 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.739091    4391 generic.go:356] "Generic (PLEG): container finished" podID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerID="2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483" exitCode=0
Oct 30 19:13:21.739247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.739132    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"
Oct 30 19:13:21.739247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.739147    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" event={"ID":"eb90e7fe-ff29-4428-a376-4b5766eac25a","Type":"ContainerDied","Data":"2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483"}
Oct 30 19:13:21.739247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.739204    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp" event={"ID":"eb90e7fe-ff29-4428-a376-4b5766eac25a","Type":"ContainerDied","Data":"3addb6387f6a4eb5320f9aaf1a7705b63ad4c8ea89974f8a8bf62dfed788a4c4"}
Oct 30 19:13:21.740919 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.740892    4391 generic.go:356] "Generic (PLEG): container finished" podID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerID="7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7" exitCode=0
Oct 30 19:13:21.741009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.740966    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" event={"ID":"e20aeee3-1a1d-43cc-879e-4f0bac0e7601","Type":"ContainerDied","Data":"7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7"}
Oct 30 19:13:21.741009 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.741001    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r" event={"ID":"e20aeee3-1a1d-43cc-879e-4f0bac0e7601","Type":"ContainerDied","Data":"cf7e77038d9f8472882776922c46e6e5335b1aa7caa91c304c5808089625a6cd"}
Oct 30 19:13:21.741096 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.741065    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"
Oct 30 19:13:21.750202 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.750182    4391 scope.go:122] "RemoveContainer" containerID="173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb"
Oct 30 19:13:21.750487 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:13:21.750460    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb\": container with ID starting with 173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb not found: ID does not exist" containerID="173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb"
Oct 30 19:13:21.750562 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.750498    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb"} err="failed to get container status \"173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb\": rpc error: code = NotFound desc = could not find container \"173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb\": container with ID starting with 173b0b36edc8d3d3c264f54eb6fea467767780986d78e16f2b3fa7bd3e98d9bb not found: ID does not exist"
Oct 30 19:13:21.750562 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.750522    4391 scope.go:122] "RemoveContainer" containerID="2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483"
Oct 30 19:13:21.760255 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.760233    4391 scope.go:122] "RemoveContainer" containerID="2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483"
Oct 30 19:13:21.760534 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:13:21.760512    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483\": container with ID starting with 2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483 not found: ID does not exist" containerID="2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483"
Oct 30 19:13:21.760621 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.760539    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483"} err="failed to get container status \"2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483\": rpc error: code = NotFound desc = could not find container \"2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483\": container with ID starting with 2e028762b57538917e0bedc5c60d05e6652f5b85281370856e551dbe779b5483 not found: ID does not exist"
Oct 30 19:13:21.760621 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.760554    4391 scope.go:122] "RemoveContainer" containerID="7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7"
Oct 30 19:13:21.769735 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.769679    4391 scope.go:122] "RemoveContainer" containerID="7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7"
Oct 30 19:13:21.769943 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:13:21.769925    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7\": container with ID starting with 7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7 not found: ID does not exist" containerID="7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7"
Oct 30 19:13:21.769988 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.769948    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7"} err="failed to get container status \"7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7\": rpc error: code = NotFound desc = could not find container \"7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7\": container with ID starting with 7848cabe9bedc12d3b5cb5369532fe3b74ca434964b685f54ad608fafd14b8c7 not found: ID does not exist"
Oct 30 19:13:21.911220 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.911148    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"]
Oct 30 19:13:21.954481 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:21.954385    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-946rp"]
Oct 30 19:13:22.042818 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.042747    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"]
Oct 30 19:13:22.047662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.047621    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-ptk7r"]
Oct 30 19:13:22.129912 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.129866    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"]
Oct 30 19:13:22.135213 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.135144    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-2wzzt"]
Oct 30 19:13:22.352763 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.352659    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"]
Oct 30 19:13:22.353460 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353437    4391 cpu_manager.go:414] "containerMap: removing container" podUID="ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757" containerName="maas-api"
Oct 30 19:13:22.353460 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353458    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757" containerName="maas-api"
Oct 30 19:13:22.353690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353484    4391 cpu_manager.go:414] "containerMap: removing container" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy"
Oct 30 19:13:22.353690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353492    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy"
Oct 30 19:13:22.353690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353514    4391 cpu_manager.go:414] "containerMap: removing container" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy"
Oct 30 19:13:22.353690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353522    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy"
Oct 30 19:13:22.353690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353544    4391 cpu_manager.go:414] "containerMap: removing container" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy"
Oct 30 19:13:22.353690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353550    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy"
Oct 30 19:13:22.353987 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353759    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" containerName="istio-proxy"
Oct 30 19:13:22.353987 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353778    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" containerName="istio-proxy"
Oct 30 19:13:22.353987 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353797    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6e4d2a-a3c0-42e0-94c0-b797fa1cc757" containerName="maas-api"
Oct 30 19:13:22.353987 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.353811    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" containerName="istio-proxy"
Oct 30 19:13:22.358967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.358946    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.408940 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.408911    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-314bd6fa-serving-cert\""
Oct 30 19:13:22.411020 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.410994    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"]
Oct 30 19:13:22.420900 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.420868    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-metrics-certs\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.421021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.421006    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dgvv\" (UniqueName: \"kubernetes.io/projected/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-kube-api-access-9dgvv\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.421086 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.421070    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-maas-api-tls\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.421309 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.421282    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-openshift-service-ca\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.523130 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.523094    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-metrics-certs\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.523292 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.523164    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dgvv\" (UniqueName: \"kubernetes.io/projected/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-kube-api-access-9dgvv\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.523292 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.523208    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-maas-api-tls\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.523404 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.523330    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-openshift-service-ca\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.523464 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:13:22.523402    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-serving-cert: secret "maas-api-e2e-mt-314bd6fa-serving-cert" not found
Oct 30 19:13:22.523519 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:13:22.523507    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-maas-api-tls podName:f045ef47-c2dc-4e15-b71e-ae1d5de0342c nodeName:}" failed. No retries permitted until 2026-10-30 19:13:23.023484818 +0000 UTC m=+2198.134218085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-maas-api-tls") pod "maas-api-e2e-mt-314bd6fa-65c588785b-8mf98" (UID: "f045ef47-c2dc-4e15-b71e-ae1d5de0342c") : secret "maas-api-e2e-mt-314bd6fa-serving-cert" not found
Oct 30 19:13:22.524013 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.523992    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-openshift-service-ca\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.525784 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.525760    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-metrics-certs\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:22.583647 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:22.583615    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dgvv\" (UniqueName: \"kubernetes.io/projected/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-kube-api-access-9dgvv\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:23.008992 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:23.008955    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20aeee3-1a1d-43cc-879e-4f0bac0e7601" path="/var/lib/kubelet/pods/e20aeee3-1a1d-43cc-879e-4f0bac0e7601/volumes"
Oct 30 19:13:23.009480 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:23.009465    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6f7fdb-32a8-479b-a026-224b18d9532f" path="/var/lib/kubelet/pods/ea6f7fdb-32a8-479b-a026-224b18d9532f/volumes"
Oct 30 19:13:23.009943 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:23.009929    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb90e7fe-ff29-4428-a376-4b5766eac25a" path="/var/lib/kubelet/pods/eb90e7fe-ff29-4428-a376-4b5766eac25a/volumes"
Oct 30 19:13:23.035455 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:23.035411    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-maas-api-tls\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:23.038062 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:23.038036    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-maas-api-tls\") pod \"maas-api-e2e-mt-314bd6fa-65c588785b-8mf98\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:23.273958 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:23.273856    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:23.649746 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:23.649716    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"]
Oct 30 19:13:23.650250 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:13:23.650223    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf045ef47_c2dc_4e15_b71e_ae1d5de0342c.slice/crio-78499693160fcebcdefbfbbafec7e62f3881b6bb5d97fa33db1163843fca1d2a WatchSource:0}: Error finding container 78499693160fcebcdefbfbbafec7e62f3881b6bb5d97fa33db1163843fca1d2a: Status 404 returned error can't find the container with id 78499693160fcebcdefbfbbafec7e62f3881b6bb5d97fa33db1163843fca1d2a
Oct 30 19:13:23.760493 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:23.760455    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98" event={"ID":"f045ef47-c2dc-4e15-b71e-ae1d5de0342c","Type":"ContainerStarted","Data":"78499693160fcebcdefbfbbafec7e62f3881b6bb5d97fa33db1163843fca1d2a"}
Oct 30 19:13:24.772734 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:24.772363    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98" event={"ID":"f045ef47-c2dc-4e15-b71e-ae1d5de0342c","Type":"ContainerStarted","Data":"bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0"}
Oct 30 19:13:24.772734 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:24.772456    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:24.883606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:24.883537    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98" podStartSLOduration=2.028404002 podStartE2EDuration="2.883520706s" totalImagesPullingTime="855.116704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:24.881602237 +0000 UTC m=+2199.992335517" watchObservedRunningTime="2026-10-30 19:13:24.883520706 +0000 UTC m=+2199.994253989"
Oct 30 19:13:30.783511 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:30.783434    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:13:32.190808 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:32.190772    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"]
Oct 30 19:13:32.191608 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:32.191553    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy" containerID="cri-o://3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072" gracePeriod=30
Oct 30 19:13:36.654260 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:36.654201    4391 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:36.654681 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:36.654292    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:37.474639 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.474615    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:13:37.555376 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555295    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-token\" (UniqueName: \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-token\") pod \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") "
Oct 30 19:13:37.555376 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555332    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istiod-ca-cert\") pod \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") "
Oct 30 19:13:37.555376 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555355    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-kube-api-access-6zzcp\" (UniqueName: \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-kube-api-access-6zzcp\") pod \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") "
Oct 30 19:13:37.555714 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555442    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-certs\") pod \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") "
Oct 30 19:13:37.555714 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555485    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-envoy\") pod \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") "
Oct 30 19:13:37.555714 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555525    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-data\") pod \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") "
Oct 30 19:13:37.555714 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555696    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-podinfo\") pod \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") "
Oct 30 19:13:37.555918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555743    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-credential-socket\") pod \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") "
Oct 30 19:13:37.555918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555779    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-socket\") pod \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\" (UID: \"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb\") "
Oct 30 19:13:37.555918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555790    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-certs" pod "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" (UID: "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:37.555918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.555860    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istiod-ca-cert" pod "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" (UID: "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:37.556123 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.556006    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-credential-socket" pod "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" (UID: "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:37.556184 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.556136    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-data" pod "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" (UID: "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:37.556184 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.556155    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-socket" pod "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" (UID: "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:37.556374 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.556347    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.556509 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.556375    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.556509 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.556401    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.556509 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.556415    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.556509 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.556428    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.557796 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.557760    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-podinfo" pod "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" (UID: "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:37.557917 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.557887    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-envoy" pod "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" (UID: "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:37.558001 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.557984    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-token" pod "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" (UID: "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:37.558044 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.557995    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-kube-api-access-6zzcp" pod "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" (UID: "79e5190c-f468-4fb9-8572-3e7ff3f7f2cb"). InnerVolumeSpecName "kube-api-access-6zzcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:37.658154 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.658115    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.658154 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.658151    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.658712 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.658165    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zzcp\" (UniqueName: \"kubernetes.io/projected/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-kube-api-access-6zzcp\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.658712 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.658180    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.862441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.862354    4391 generic.go:356] "Generic (PLEG): container finished" podID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerID="3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072" exitCode=0
Oct 30 19:13:37.862441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.862399    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"
Oct 30 19:13:37.862684 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.862435    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" event={"ID":"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb","Type":"ContainerDied","Data":"3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072"}
Oct 30 19:13:37.862684 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.862489    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s" event={"ID":"79e5190c-f468-4fb9-8572-3e7ff3f7f2cb","Type":"ContainerDied","Data":"043cfe463f92b8f4e92cd47dc1ab7c6d45fe5b9901dd63742f16eea45b80224a"}
Oct 30 19:13:37.862684 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.862515    4391 scope.go:122] "RemoveContainer" containerID="3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072"
Oct 30 19:13:37.873781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.873765    4391 scope.go:122] "RemoveContainer" containerID="3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072"
Oct 30 19:13:37.874061 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:13:37.874043    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072\": container with ID starting with 3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072 not found: ID does not exist" containerID="3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072"
Oct 30 19:13:37.874134 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:37.874069    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072"} err="failed to get container status \"3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072\": rpc error: code = NotFound desc = could not find container \"3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072\": container with ID starting with 3f5f8136884ae65767dbcc2ce6468c6d0662d22c39e9e988deb42d4a536be072 not found: ID does not exist"
Oct 30 19:13:38.022167 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.022127    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"]
Oct 30 19:13:38.029037 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.029010    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-hvl9s"]
Oct 30 19:13:38.100122 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.100082    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"]
Oct 30 19:13:38.100357 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.100316    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj" podUID="79564581-842c-4551-8880-b31c78cd608e" containerName="maas-api" containerID="cri-o://b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97" gracePeriod=30
Oct 30 19:13:38.457457 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.457430    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:13:38.585557 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.584136    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-maas-api-tls\") pod \"79564581-842c-4551-8880-b31c78cd608e\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") "
Oct 30 19:13:38.585557 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.584239    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79564581-842c-4551-8880-b31c78cd608e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79564581-842c-4551-8880-b31c78cd608e-openshift-service-ca\") pod \"79564581-842c-4551-8880-b31c78cd608e\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") "
Oct 30 19:13:38.585557 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.584309    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79564581-842c-4551-8880-b31c78cd608e-kube-api-access-bdlhk\" (UniqueName: \"kubernetes.io/projected/79564581-842c-4551-8880-b31c78cd608e-kube-api-access-bdlhk\") pod \"79564581-842c-4551-8880-b31c78cd608e\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") "
Oct 30 19:13:38.585557 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.584411    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-metrics-certs\") pod \"79564581-842c-4551-8880-b31c78cd608e\" (UID: \"79564581-842c-4551-8880-b31c78cd608e\") "
Oct 30 19:13:38.585557 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.585382    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79564581-842c-4551-8880-b31c78cd608e-openshift-service-ca" pod "79564581-842c-4551-8880-b31c78cd608e" (UID: "79564581-842c-4551-8880-b31c78cd608e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:38.600742 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.600693    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-metrics-certs" pod "79564581-842c-4551-8880-b31c78cd608e" (UID: "79564581-842c-4551-8880-b31c78cd608e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:38.602789 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.602740    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-maas-api-tls" pod "79564581-842c-4551-8880-b31c78cd608e" (UID: "79564581-842c-4551-8880-b31c78cd608e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:38.602789 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.602764    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79564581-842c-4551-8880-b31c78cd608e-kube-api-access-bdlhk" pod "79564581-842c-4551-8880-b31c78cd608e" (UID: "79564581-842c-4551-8880-b31c78cd608e"). InnerVolumeSpecName "kube-api-access-bdlhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:38.686036 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.685998    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdlhk\" (UniqueName: \"kubernetes.io/projected/79564581-842c-4551-8880-b31c78cd608e-kube-api-access-bdlhk\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:38.686036 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.686032    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:38.686036 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.686042    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79564581-842c-4551-8880-b31c78cd608e-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:38.686500 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.686051    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79564581-842c-4551-8880-b31c78cd608e-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:38.872373 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.872274    4391 generic.go:356] "Generic (PLEG): container finished" podID="79564581-842c-4551-8880-b31c78cd608e" containerID="b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97" exitCode=0
Oct 30 19:13:38.872373 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.872329    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"
Oct 30 19:13:38.872373 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.872361    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj" event={"ID":"79564581-842c-4551-8880-b31c78cd608e","Type":"ContainerDied","Data":"b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97"}
Oct 30 19:13:38.872699 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.872412    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj" event={"ID":"79564581-842c-4551-8880-b31c78cd608e","Type":"ContainerDied","Data":"7c49947b53560c175f4bad41df3b8132f1d63da6d2feae7123e88acea5b99dec"}
Oct 30 19:13:38.872699 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.872435    4391 scope.go:122] "RemoveContainer" containerID="b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97"
Oct 30 19:13:38.883192 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.883175    4391 scope.go:122] "RemoveContainer" containerID="b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97"
Oct 30 19:13:38.883446 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:13:38.883421    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97\": container with ID starting with b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97 not found: ID does not exist" containerID="b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97"
Oct 30 19:13:38.883502 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:38.883456    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97"} err="failed to get container status \"b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97\": rpc error: code = NotFound desc = could not find container \"b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97\": container with ID starting with b0bed6f53d518e31228fe5b7c83a4f6bb0844a38f00a352bd3d2778a507b5a97 not found: ID does not exist"
Oct 30 19:13:39.008354 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:39.008325    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" path="/var/lib/kubelet/pods/79e5190c-f468-4fb9-8572-3e7ff3f7f2cb/volumes"
Oct 30 19:13:39.033689 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:39.033653    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"]
Oct 30 19:13:39.037755 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:39.037733    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3bd578-cc8894c49-np8pj"]
Oct 30 19:13:41.014937 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:41.014901    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79564581-842c-4551-8880-b31c78cd608e" path="/var/lib/kubelet/pods/79564581-842c-4551-8880-b31c78cd608e/volumes"
Oct 30 19:13:41.253701 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:13:41.253665    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:13:54.927436 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:54.927401    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"]
Oct 30 19:13:54.928163 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:54.927657    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx" podUID="34b7732a-b154-4fdd-a7a9-e8f81b734a77" containerName="maas-api" containerID="cri-o://799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0" gracePeriod=30
Oct 30 19:13:55.289912 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.289870    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:13:55.368486 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.368444    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b7732a-b154-4fdd-a7a9-e8f81b734a77-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b7732a-b154-4fdd-a7a9-e8f81b734a77-openshift-service-ca\") pod \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") "
Oct 30 19:13:55.368686 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.368546    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-maas-api-tls\") pod \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") "
Oct 30 19:13:55.368686 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.368671    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b7732a-b154-4fdd-a7a9-e8f81b734a77-kube-api-access-tvc4w\" (UniqueName: \"kubernetes.io/projected/34b7732a-b154-4fdd-a7a9-e8f81b734a77-kube-api-access-tvc4w\") pod \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") "
Oct 30 19:13:55.368826 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.368809    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-metrics-certs\") pod \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\" (UID: \"34b7732a-b154-4fdd-a7a9-e8f81b734a77\") "
Oct 30 19:13:55.368941 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.368912    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b7732a-b154-4fdd-a7a9-e8f81b734a77-openshift-service-ca" pod "34b7732a-b154-4fdd-a7a9-e8f81b734a77" (UID: "34b7732a-b154-4fdd-a7a9-e8f81b734a77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:55.369369 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.369344    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34b7732a-b154-4fdd-a7a9-e8f81b734a77-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:55.370867 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.370840    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-metrics-certs" pod "34b7732a-b154-4fdd-a7a9-e8f81b734a77" (UID: "34b7732a-b154-4fdd-a7a9-e8f81b734a77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:55.370995 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.370972    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b7732a-b154-4fdd-a7a9-e8f81b734a77-kube-api-access-tvc4w" pod "34b7732a-b154-4fdd-a7a9-e8f81b734a77" (UID: "34b7732a-b154-4fdd-a7a9-e8f81b734a77"). InnerVolumeSpecName "kube-api-access-tvc4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:55.371185 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.371167    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-maas-api-tls" pod "34b7732a-b154-4fdd-a7a9-e8f81b734a77" (UID: "34b7732a-b154-4fdd-a7a9-e8f81b734a77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:55.474918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.471665    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:55.474918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.471708    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvc4w\" (UniqueName: \"kubernetes.io/projected/34b7732a-b154-4fdd-a7a9-e8f81b734a77-kube-api-access-tvc4w\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:55.474918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:55.471726    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34b7732a-b154-4fdd-a7a9-e8f81b734a77-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:56.088243 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:56.088201    4391 generic.go:356] "Generic (PLEG): container finished" podID="34b7732a-b154-4fdd-a7a9-e8f81b734a77" containerID="799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0" exitCode=0
Oct 30 19:13:56.088773 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:56.088247    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx" event={"ID":"34b7732a-b154-4fdd-a7a9-e8f81b734a77","Type":"ContainerDied","Data":"799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0"}
Oct 30 19:13:56.088773 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:56.088305    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"
Oct 30 19:13:56.088773 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:56.088322    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx" event={"ID":"34b7732a-b154-4fdd-a7a9-e8f81b734a77","Type":"ContainerDied","Data":"d06aab6a44c8d054a280569f52fa85921f5adba08423d4d582a483ab62dbdce3"}
Oct 30 19:13:56.088773 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:56.088356    4391 scope.go:122] "RemoveContainer" containerID="799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0"
Oct 30 19:13:56.100544 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:56.100526    4391 scope.go:122] "RemoveContainer" containerID="799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0"
Oct 30 19:13:56.100851 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:13:56.100830    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0\": container with ID starting with 799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0 not found: ID does not exist" containerID="799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0"
Oct 30 19:13:56.100929 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:56.100860    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0"} err="failed to get container status \"799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0\": rpc error: code = NotFound desc = could not find container \"799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0\": container with ID starting with 799807a9d8d6c4ce6b895103bc94220172f71df178f0ff56c39390f238d576e0 not found: ID does not exist"
Oct 30 19:13:56.248472 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:56.248440    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"]
Oct 30 19:13:56.294703 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:56.294662    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8be536-5446776ff4-hxpjx"]
Oct 30 19:13:57.007268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:57.007231    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b7732a-b154-4fdd-a7a9-e8f81b734a77" path="/var/lib/kubelet/pods/34b7732a-b154-4fdd-a7a9-e8f81b734a77/volumes"
Oct 30 19:13:59.700273 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:13:59.700232    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"]
Oct 30 19:14:04.700801 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:04.700493    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr" podUID="b3868727-94fa-41b9-bd83-a9ff0c537971" containerName="payload-pre-processing" containerID="cri-o://03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc" gracePeriod=25
Oct 30 19:14:05.207654 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.207554    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:14:05.233721 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.233674    4391 generic.go:356] "Generic (PLEG): container finished" podID="b3868727-94fa-41b9-bd83-a9ff0c537971" containerID="03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc" exitCode=0
Oct 30 19:14:05.234128 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.234104    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"
Oct 30 19:14:05.234359 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.234324    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr" event={"ID":"b3868727-94fa-41b9-bd83-a9ff0c537971","Type":"ContainerDied","Data":"03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc"}
Oct 30 19:14:05.234602 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.234562    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr" event={"ID":"b3868727-94fa-41b9-bd83-a9ff0c537971","Type":"ContainerDied","Data":"010159b71fb8c0c40fa920a8b4401fed348f64807794e8966d83954b0d204cbe"}
Oct 30 19:14:05.234872 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.234624    4391 scope.go:122] "RemoveContainer" containerID="03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc"
Oct 30 19:14:05.245858 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.245836    4391 scope.go:122] "RemoveContainer" containerID="03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc"
Oct 30 19:14:05.246174 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:05.246153    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc\": container with ID starting with 03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc not found: ID does not exist" containerID="03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc"
Oct 30 19:14:05.246237 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.246187    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc"} err="failed to get container status \"03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc\": rpc error: code = NotFound desc = could not find container \"03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc\": container with ID starting with 03d970e8b689224d132a56b74f322b606247013bc0e0ea93fe44fb72a0078fcc not found: ID does not exist"
Oct 30 19:14:05.328035 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.327947    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3868727-94fa-41b9-bd83-a9ff0c537971-kube-api-access-lnmf9\" (UniqueName: \"kubernetes.io/projected/b3868727-94fa-41b9-bd83-a9ff0c537971-kube-api-access-lnmf9\") pod \"b3868727-94fa-41b9-bd83-a9ff0c537971\" (UID: \"b3868727-94fa-41b9-bd83-a9ff0c537971\") "
Oct 30 19:14:05.328189 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.328073    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3868727-94fa-41b9-bd83-a9ff0c537971-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3868727-94fa-41b9-bd83-a9ff0c537971-plugins-config-volume\") pod \"b3868727-94fa-41b9-bd83-a9ff0c537971\" (UID: \"b3868727-94fa-41b9-bd83-a9ff0c537971\") "
Oct 30 19:14:05.328491 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.328458    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3868727-94fa-41b9-bd83-a9ff0c537971-plugins-config-volume" pod "b3868727-94fa-41b9-bd83-a9ff0c537971" (UID: "b3868727-94fa-41b9-bd83-a9ff0c537971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:05.328760 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.328737    4391 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3868727-94fa-41b9-bd83-a9ff0c537971-plugins-config-volume\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:05.330278 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.330255    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3868727-94fa-41b9-bd83-a9ff0c537971-kube-api-access-lnmf9" pod "b3868727-94fa-41b9-bd83-a9ff0c537971" (UID: "b3868727-94fa-41b9-bd83-a9ff0c537971"). InnerVolumeSpecName "kube-api-access-lnmf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:05.430417 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.430382    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmf9\" (UniqueName: \"kubernetes.io/projected/b3868727-94fa-41b9-bd83-a9ff0c537971-kube-api-access-lnmf9\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:05.651655 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.651487    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"]
Oct 30 19:14:05.657689 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:05.657658    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-091e8f-5d5864d5dc-rp9lr"]
Oct 30 19:14:07.007240 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:07.007203    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3868727-94fa-41b9-bd83-a9ff0c537971" path="/var/lib/kubelet/pods/b3868727-94fa-41b9-bd83-a9ff0c537971/volumes"
Oct 30 19:14:11.253070 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:11.253033    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:14:12.278335 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:12.278298    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:14:23.555015 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:23.554978    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"]
Oct 30 19:14:23.555659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:23.555245    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf" podUID="4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1" containerName="maas-api" containerID="cri-o://c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d" gracePeriod=30
Oct 30 19:14:23.999617 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:23.997797    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.040339    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-metrics-certs\") pod \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") "
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.040458    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-maas-api-tls\") pod \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") "
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.040542    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-openshift-service-ca\") pod \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") "
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.040674    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-kube-api-access-t5pxl\" (UniqueName: \"kubernetes.io/projected/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-kube-api-access-t5pxl\") pod \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\" (UID: \"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1\") "
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.051508    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-metrics-certs" pod "4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1" (UID: "4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.063951    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-openshift-service-ca" pod "4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1" (UID: "4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.067910    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-maas-api-tls" pod "4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1" (UID: "4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.068861    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-kube-api-access-t5pxl" pod "4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1" (UID: "4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1"). InnerVolumeSpecName "kube-api-access-t5pxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.079080    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.079170    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5pxl\" (UniqueName: \"kubernetes.io/projected/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-kube-api-access-t5pxl\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.079188    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:24.098403 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.079204    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:24.377323 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.377237    4391 generic.go:356] "Generic (PLEG): container finished" podID="4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1" containerID="c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d" exitCode=0
Oct 30 19:14:24.377323 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.377288    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"
Oct 30 19:14:24.377518 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.377320    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf" event={"ID":"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1","Type":"ContainerDied","Data":"c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d"}
Oct 30 19:14:24.377518 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.377378    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf" event={"ID":"4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1","Type":"ContainerDied","Data":"d6713fbc79fe61af4279eff0aca759de9fe98b7b200a1fab5c6f1739f91bda40"}
Oct 30 19:14:24.377518 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.377401    4391 scope.go:122] "RemoveContainer" containerID="c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d"
Oct 30 19:14:24.388497 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.388479    4391 scope.go:122] "RemoveContainer" containerID="c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d"
Oct 30 19:14:24.388780 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:24.388763    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d\": container with ID starting with c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d not found: ID does not exist" containerID="c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d"
Oct 30 19:14:24.388847 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.388793    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d"} err="failed to get container status \"c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d\": rpc error: code = NotFound desc = could not find container \"c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d\": container with ID starting with c8e56cd592d7862f3d1c3f9b77551bb92f802503799deb4ad8607810088b931d not found: ID does not exist"
Oct 30 19:14:24.536190 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.536150    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"]
Oct 30 19:14:24.596463 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:24.596415    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0f3f2f-747f4d4565-88xxf"]
Oct 30 19:14:25.007087 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:25.007051    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1" path="/var/lib/kubelet/pods/4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1/volumes"
Oct 30 19:14:33.410442 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.410396    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"]
Oct 30 19:14:33.411067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411050    4391 cpu_manager.go:414] "containerMap: removing container" podUID="79564581-842c-4551-8880-b31c78cd608e" containerName="maas-api"
Oct 30 19:14:33.411067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411064    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79564581-842c-4551-8880-b31c78cd608e" containerName="maas-api"
Oct 30 19:14:33.411067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411092    4391 cpu_manager.go:414] "containerMap: removing container" podUID="34b7732a-b154-4fdd-a7a9-e8f81b734a77" containerName="maas-api"
Oct 30 19:14:33.411067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411097    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b7732a-b154-4fdd-a7a9-e8f81b734a77" containerName="maas-api"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411111    4391 cpu_manager.go:414] "containerMap: removing container" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411117    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411133    4391 cpu_manager.go:414] "containerMap: removing container" podUID="4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1" containerName="maas-api"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411138    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1" containerName="maas-api"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411148    4391 cpu_manager.go:414] "containerMap: removing container" podUID="b3868727-94fa-41b9-bd83-a9ff0c537971" containerName="payload-pre-processing"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411152    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3868727-94fa-41b9-bd83-a9ff0c537971" containerName="payload-pre-processing"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411240    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="79564581-842c-4551-8880-b31c78cd608e" containerName="maas-api"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411257    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b7732a-b154-4fdd-a7a9-e8f81b734a77" containerName="maas-api"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411266    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e5190c-f468-4fb9-8572-3e7ff3f7f2cb" containerName="istio-proxy"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411274    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5636d2-9ff1-4fe0-a28e-3eb60a07e0d1" containerName="maas-api"
Oct 30 19:14:33.411321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.411280    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3868727-94fa-41b9-bd83-a9ff0c537971" containerName="payload-pre-processing"
Oct 30 19:14:33.415999 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.415979    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.465430 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.465404    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-314bd6fa-openshift-default-dockercfg-rtl22\""
Oct 30 19:14:33.520433 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.520395    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.520617 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.520564    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.520684 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.520668    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwhd6\" (UniqueName: \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-kube-api-access-kwhd6\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.520724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.520690    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.520785 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.520739    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.520829 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.520817    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.520911 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.520892    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.520973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.520928    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52d0536-fca2-49cf-b0af-9d78f5b14829-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.520973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.520946    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.528501 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.528471    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"]
Oct 30 19:14:33.621861 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.621826    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622051 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.621871    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622051 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.621927    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622186 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622059    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52d0536-fca2-49cf-b0af-9d78f5b14829-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622186 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622107    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622295 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622193    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622295 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622278    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622420 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622397    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622485 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622469    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwhd6\" (UniqueName: \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-kube-api-access-kwhd6\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622541 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622494    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622541 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622499    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622672 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622608    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622900 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622876    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52d0536-fca2-49cf-b0af-9d78f5b14829-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.622998 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.622976    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.625165 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.625137    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.625561 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.625538    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.678902 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.678824    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.679055 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.679030    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwhd6\" (UniqueName: \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-kube-api-access-kwhd6\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:33.730532 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:33.730504    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:34.186236 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:34.186206    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"]
Oct 30 19:14:34.188089 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:14:34.188056    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52d0536_fca2_49cf_b0af_9d78f5b14829.slice/crio-27d47626829c51a3a5fecfebde039dd69eb9b16f265bff546a40083aadf7c5a0 WatchSource:0}: Error finding container 27d47626829c51a3a5fecfebde039dd69eb9b16f265bff546a40083aadf7c5a0: Status 404 returned error can't find the container with id 27d47626829c51a3a5fecfebde039dd69eb9b16f265bff546a40083aadf7c5a0
Oct 30 19:14:34.204453 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:34.191840    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:34.204453 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:34.191939    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:34.204453 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:34.191980    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:34.445411 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:34.445264    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" event={"ID":"c52d0536-fca2-49cf-b0af-9d78f5b14829","Type":"ContainerStarted","Data":"2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6"}
Oct 30 19:14:34.445411 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:34.445330    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" event={"ID":"c52d0536-fca2-49cf-b0af-9d78f5b14829","Type":"ContainerStarted","Data":"27d47626829c51a3a5fecfebde039dd69eb9b16f265bff546a40083aadf7c5a0"}
Oct 30 19:14:34.563386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:34.563311    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" podStartSLOduration=1.563292529 podStartE2EDuration="1.563292529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:34.559866709 +0000 UTC m=+2269.670600007" watchObservedRunningTime="2026-10-30 19:14:34.563292529 +0000 UTC m=+2269.674025814"
Oct 30 19:14:34.731303 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:34.731216    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:35.733091 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:35.733050    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:14:35.733551 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:35.733116    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" podUID="c52d0536-fca2-49cf-b0af-9d78f5b14829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 19:14:36.730996 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:36.730959    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:14:36.731207 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:36.731031    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" podUID="c52d0536-fca2-49cf-b0af-9d78f5b14829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 19:14:37.731004 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:37.730973    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:14:37.731382 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:37.731057    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" podUID="c52d0536-fca2-49cf-b0af-9d78f5b14829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 19:14:38.730863 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:38.730828    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:14:38.731037 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:38.730886    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" podUID="c52d0536-fca2-49cf-b0af-9d78f5b14829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 19:14:39.545392 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:39.545349    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"]
Oct 30 19:14:39.592611 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:39.592368    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"]
Oct 30 19:14:39.595740 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:39.595709    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"]
Oct 30 19:14:39.731798 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:39.731767    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:14:39.732225 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:39.731821    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" podUID="c52d0536-fca2-49cf-b0af-9d78f5b14829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 19:14:40.859275 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:40.859241    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:40.859827 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:40.859659    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:40.860677 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:40.860653    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:14:41.300407 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:41.300367    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:14:48.511979 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.511939    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"]
Oct 30 19:14:48.517661 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.517641    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.626449 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.626411    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"]
Oct 30 19:14:48.655542 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.655508    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.655542 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.655548    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.655800 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.655568    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.655800 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.655684    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.655800 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.655707    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqkxq\" (UniqueName: \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-kube-api-access-lqkxq\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.655914 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.655828    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.655957 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.655900    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.656060 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.656042    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.656169 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.656154    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.757898 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.757850    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758121 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.757972    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758121 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758005    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758121 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758034    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758121 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758059    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758121 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758085    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqkxq\" (UniqueName: \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-kube-api-access-lqkxq\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758396 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758204    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758396 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758233    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758396 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758343    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758548 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758471    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758703 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758641    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758703 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758675    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.758703 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.758701    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.759092 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.759069    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.760278 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.760258    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.760750 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.760727    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.813238 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.813161    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.813398 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.813161    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqkxq\" (UniqueName: \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-kube-api-access-lqkxq\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:48.835186 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:48.835143    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:49.308405 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:49.308368    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"]
Oct 30 19:14:49.344620 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:14:49.344553    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334efd0f_aefb_4c27_baf9_f8e51ef0b6ec.slice/crio-33dfda68ec4e8f45f047cd6aa0e2e48cea3f2e7ec51fb54be97f0dbccca5f373 WatchSource:0}: Error finding container 33dfda68ec4e8f45f047cd6aa0e2e48cea3f2e7ec51fb54be97f0dbccca5f373: Status 404 returned error can't find the container with id 33dfda68ec4e8f45f047cd6aa0e2e48cea3f2e7ec51fb54be97f0dbccca5f373
Oct 30 19:14:49.347306 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:49.347273    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:49.347390 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:49.347344    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:49.347390 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:49.347374    4391 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:49.574456 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:49.574362    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" event={"ID":"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec","Type":"ContainerStarted","Data":"44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae"}
Oct 30 19:14:49.574456 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:49.574422    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" event={"ID":"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec","Type":"ContainerStarted","Data":"33dfda68ec4e8f45f047cd6aa0e2e48cea3f2e7ec51fb54be97f0dbccca5f373"}
Oct 30 19:14:49.687799 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:49.687733    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" podStartSLOduration=1.6877152 podStartE2EDuration="1.6877152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:49.684426459 +0000 UTC m=+2284.795159747" watchObservedRunningTime="2026-10-30 19:14:49.6877152 +0000 UTC m=+2284.798448484"
Oct 30 19:14:49.836666 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:49.835687    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:50.236616 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.236553    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8873-29562a"]
Oct 30 19:14:50.242557 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.242526    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:14:50.274647 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.261914    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8873-29562a"]
Oct 30 19:14:50.296085 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.296043    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qk8z9\""
Oct 30 19:14:50.392707 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.392668    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8kr\" (UniqueName: \"kubernetes.io/projected/02d77b72-1171-4501-86bb-415af15ffaad-kube-api-access-5l8kr\") pod \"test-curl-8873-29562a\" (UID: \"02d77b72-1171-4501-86bb-415af15ffaad\") " pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:14:50.493662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.493533    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8kr\" (UniqueName: \"kubernetes.io/projected/02d77b72-1171-4501-86bb-415af15ffaad-kube-api-access-5l8kr\") pod \"test-curl-8873-29562a\" (UID: \"02d77b72-1171-4501-86bb-415af15ffaad\") " pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:14:50.558603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.558168    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8kr\" (UniqueName: \"kubernetes.io/projected/02d77b72-1171-4501-86bb-415af15ffaad-kube-api-access-5l8kr\") pod \"test-curl-8873-29562a\" (UID: \"02d77b72-1171-4501-86bb-415af15ffaad\") " pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:14:50.576465 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.576430    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:14:50.837166 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.837084    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:14:50.837307 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.837155    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 19:14:50.976227 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:50.976189    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8873-29562a"]
Oct 30 19:14:50.976941 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:14:50.976910    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02d77b72_1171_4501_86bb_415af15ffaad.slice/crio-7d189f907d5096e836df1e0d077b978da4494c36f7f14c5845210ce8cb1c5c56 WatchSource:0}: Error finding container 7d189f907d5096e836df1e0d077b978da4494c36f7f14c5845210ce8cb1c5c56: Status 404 returned error can't find the container with id 7d189f907d5096e836df1e0d077b978da4494c36f7f14c5845210ce8cb1c5c56
Oct 30 19:14:51.589519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:51.589485    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8873-29562a" event={"ID":"02d77b72-1171-4501-86bb-415af15ffaad","Type":"ContainerStarted","Data":"010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299"}
Oct 30 19:14:51.589940 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:51.589529    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8873-29562a" event={"ID":"02d77b72-1171-4501-86bb-415af15ffaad","Type":"ContainerStarted","Data":"7d189f907d5096e836df1e0d077b978da4494c36f7f14c5845210ce8cb1c5c56"}
Oct 30 19:14:51.698870 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:51.698813    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8873-29562a" podStartSLOduration=1.698800078 podStartE2EDuration="1.698800078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:51.695689883 +0000 UTC m=+2286.806423164" watchObservedRunningTime="2026-10-30 19:14:51.698800078 +0000 UTC m=+2286.809533360"
Oct 30 19:14:51.835847 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:51.835813    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:14:51.836032 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:51.835872    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 19:14:52.835728 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:52.835691    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:14:52.836138 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:52.835747    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 19:14:53.332603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:53.327314    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8873-29562a"]
Oct 30 19:14:53.360603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:53.355385    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8873-29562a"]
Oct 30 19:14:53.602769 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:53.602679    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8873-29562a" podUID="02d77b72-1171-4501-86bb-415af15ffaad" containerName="test-curl-8873-29562a" containerID="cri-o://010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299" gracePeriod=2
Oct 30 19:14:53.666651 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:53.665873    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8873-29562a\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="02d77b72-1171-4501-86bb-415af15ffaad" pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:14:53.836391 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:53.836353    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:14:53.836809 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:53.836421    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 19:14:54.593124 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:54.593084    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" podUID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" containerName="main" containerID="cri-o://9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b" gracePeriod=45
Oct 30 19:14:54.613887 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:54.613850    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" containerID="cri-o://ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef" gracePeriod=45
Oct 30 19:14:54.616951 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:54.616907    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" containerID="cri-o://0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346" gracePeriod=45
Oct 30 19:14:54.838441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:54.836938    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:14:54.838441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:54.836994    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 19:14:54.838441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:54.837367    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:14:55.030206 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.030002    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 19:14:55.032990 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.032944    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:14:55.040442 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.040422    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 19:14:55.066861 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:55.066821    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8873-29562a\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="02d77b72-1171-4501-86bb-415af15ffaad" pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:14:55.188115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188079    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tmp-dir\") pod \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") "
Oct 30 19:14:55.188115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188124    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-kube-api-access-qnd5f\" (UniqueName: \"kubernetes.io/projected/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-kube-api-access-qnd5f\") pod \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") "
Oct 30 19:14:55.188402 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188163    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-model-cache\") pod \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") "
Oct 30 19:14:55.188402 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188219    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a69d375-f817-4870-951d-e6969a279191-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a69d375-f817-4870-951d-e6969a279191-tls-certs\") pod \"0a69d375-f817-4870-951d-e6969a279191\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") "
Oct 30 19:14:55.188402 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188259    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a69d375-f817-4870-951d-e6969a279191-kube-api-access-n5w7c\" (UniqueName: \"kubernetes.io/projected/0a69d375-f817-4870-951d-e6969a279191-kube-api-access-n5w7c\") pod \"0a69d375-f817-4870-951d-e6969a279191\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") "
Oct 30 19:14:55.188402 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188295    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-tmp-dir\") pod \"0a69d375-f817-4870-951d-e6969a279191\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") "
Oct 30 19:14:55.188402 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188327    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-model-cache\") pod \"0a69d375-f817-4870-951d-e6969a279191\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") "
Oct 30 19:14:55.188402 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188378    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-home\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-home\") pod \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") "
Oct 30 19:14:55.188781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188429    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tmp-dir\") pod \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") "
Oct 30 19:14:55.188781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188523    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-home\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-home\") pod \"0a69d375-f817-4870-951d-e6969a279191\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") "
Oct 30 19:14:55.188781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188549    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-home\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-home\") pod \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") "
Oct 30 19:14:55.188781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188601    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-model-cache\") pod \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") "
Oct 30 19:14:55.188781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188662    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tls-certs\") pod \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") "
Oct 30 19:14:55.188781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188696    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-dshm\") pod \"0a69d375-f817-4870-951d-e6969a279191\" (UID: \"0a69d375-f817-4870-951d-e6969a279191\") "
Oct 30 19:14:55.188781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188731    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tls-certs\") pod \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") "
Oct 30 19:14:55.189136 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188807    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-dshm\") pod \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\" (UID: \"58ba6f26-346c-4cfa-8af2-5b46713e3c6e\") "
Oct 30 19:14:55.189136 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188843    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-kube-api-access-m9spb\" (UniqueName: \"kubernetes.io/projected/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-kube-api-access-m9spb\") pod \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") "
Oct 30 19:14:55.189136 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188856    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-model-cache" pod "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" (UID: "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.189136 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.188872    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-dshm\") pod \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\" (UID: \"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf\") "
Oct 30 19:14:55.189526 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.189464    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-tmp-dir" pod "0a69d375-f817-4870-951d-e6969a279191" (UID: "0a69d375-f817-4870-951d-e6969a279191"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.189658 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.189616    4391 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-model-cache\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.189789 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.189764    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tmp-dir" pod "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" (UID: "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.190075 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.190048    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-model-cache" pod "0a69d375-f817-4870-951d-e6969a279191" (UID: "0a69d375-f817-4870-951d-e6969a279191"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.191084 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.191055    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-home" pod "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" (UID: "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.191562 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.191195    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tmp-dir" pod "58ba6f26-346c-4cfa-8af2-5b46713e3c6e" (UID: "58ba6f26-346c-4cfa-8af2-5b46713e3c6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.191562 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.191210    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-home" pod "0a69d375-f817-4870-951d-e6969a279191" (UID: "0a69d375-f817-4870-951d-e6969a279191"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.191562 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.191480    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-model-cache" pod "58ba6f26-346c-4cfa-8af2-5b46713e3c6e" (UID: "58ba6f26-346c-4cfa-8af2-5b46713e3c6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.191857 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.191677    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-home" pod "58ba6f26-346c-4cfa-8af2-5b46713e3c6e" (UID: "58ba6f26-346c-4cfa-8af2-5b46713e3c6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.191857 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.191828    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-dshm" pod "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" (UID: "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.195015 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.194981    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a69d375-f817-4870-951d-e6969a279191-kube-api-access-n5w7c" pod "0a69d375-f817-4870-951d-e6969a279191" (UID: "0a69d375-f817-4870-951d-e6969a279191"). InnerVolumeSpecName "kube-api-access-n5w7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:55.195739 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.195678    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-dshm" pod "58ba6f26-346c-4cfa-8af2-5b46713e3c6e" (UID: "58ba6f26-346c-4cfa-8af2-5b46713e3c6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.195739 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.195707    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a69d375-f817-4870-951d-e6969a279191-tls-certs" pod "0a69d375-f817-4870-951d-e6969a279191" (UID: "0a69d375-f817-4870-951d-e6969a279191"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:55.195975 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.195727    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tls-certs" pod "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" (UID: "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:55.196095 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.196057    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tls-certs" pod "58ba6f26-346c-4cfa-8af2-5b46713e3c6e" (UID: "58ba6f26-346c-4cfa-8af2-5b46713e3c6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:55.196213 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.196190    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-dshm" pod "0a69d375-f817-4870-951d-e6969a279191" (UID: "0a69d375-f817-4870-951d-e6969a279191"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.197154 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.197131    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-kube-api-access-m9spb" pod "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" (UID: "8bc53cc7-83c4-495b-8176-a15ef5ce4ecf"). InnerVolumeSpecName "kube-api-access-m9spb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:55.206076 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.206047    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-kube-api-access-qnd5f" pod "58ba6f26-346c-4cfa-8af2-5b46713e3c6e" (UID: "58ba6f26-346c-4cfa-8af2-5b46713e3c6e"). InnerVolumeSpecName "kube-api-access-qnd5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:55.290794 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290761    4391 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-dshm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.290794 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290789    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9spb\" (UniqueName: \"kubernetes.io/projected/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-kube-api-access-m9spb\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.290794 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290799    4391 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-dshm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290807    4391 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tmp-dir\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290815    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnd5f\" (UniqueName: \"kubernetes.io/projected/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-kube-api-access-qnd5f\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290825    4391 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a69d375-f817-4870-951d-e6969a279191-tls-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290834    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5w7c\" (UniqueName: \"kubernetes.io/projected/0a69d375-f817-4870-951d-e6969a279191-kube-api-access-n5w7c\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290842    4391 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-tmp-dir\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290850    4391 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-model-cache\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290875    4391 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-home\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290884    4391 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tmp-dir\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290892    4391 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-home\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290899    4391 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-home\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290908    4391 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-model-cache\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290916    4391 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf-tls-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290923    4391 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a69d375-f817-4870-951d-e6969a279191-dshm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.291386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.290930    4391 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58ba6f26-346c-4cfa-8af2-5b46713e3c6e-tls-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.626272 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.626249    4391 generic.go:356] "Generic (PLEG): container finished" podID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerID="0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346" exitCode=0
Oct 30 19:14:55.629677 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.626464    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"
Oct 30 19:14:55.629677 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.626485    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" event={"ID":"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf","Type":"ContainerDied","Data":"0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346"}
Oct 30 19:14:55.629677 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.628899    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx" event={"ID":"8bc53cc7-83c4-495b-8176-a15ef5ce4ecf","Type":"ContainerDied","Data":"15bb42fc022ef4614d1cd54bc377f497490d70b71a32f42c511d1601518a0b6d"}
Oct 30 19:14:55.629677 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.628931    4391 scope.go:122] "RemoveContainer" containerID="0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346"
Oct 30 19:14:55.635766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.633187    4391 generic.go:356] "Generic (PLEG): container finished" podID="0a69d375-f817-4870-951d-e6969a279191" containerID="ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef" exitCode=0
Oct 30 19:14:55.635766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.633288    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" event={"ID":"0a69d375-f817-4870-951d-e6969a279191","Type":"ContainerDied","Data":"ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef"}
Oct 30 19:14:55.635766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.633312    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" event={"ID":"0a69d375-f817-4870-951d-e6969a279191","Type":"ContainerDied","Data":"5c093e525054c5bcfe7de683e57d03c62f2e180db51cf5d996eb3848ec7437fc"}
Oct 30 19:14:55.635766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.633389    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"
Oct 30 19:14:55.644911 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.644537    4391 generic.go:356] "Generic (PLEG): container finished" podID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" containerID="9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b" exitCode=0
Oct 30 19:14:55.644911 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.644634    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" event={"ID":"58ba6f26-346c-4cfa-8af2-5b46713e3c6e","Type":"ContainerDied","Data":"9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b"}
Oct 30 19:14:55.644911 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.644645    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"
Oct 30 19:14:55.644911 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.644662    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp" event={"ID":"58ba6f26-346c-4cfa-8af2-5b46713e3c6e","Type":"ContainerDied","Data":"47e501f68fce224cd13d58fc99c60f6c8b1b07cd1762f41bdd8b12323e4ba984"}
Oct 30 19:14:55.717669 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.717647    4391 scope.go:122] "RemoveContainer" containerID="0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346"
Oct 30 19:14:55.718019 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:55.717996    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346\": container with ID starting with 0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346 not found: ID does not exist" containerID="0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346"
Oct 30 19:14:55.718074 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.718028    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346"} err="failed to get container status \"0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346\": rpc error: code = NotFound desc = could not find container \"0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346\": container with ID starting with 0dddf350ac639fdbeb4b57b21688f51f7f059b435203716d3be97d95322bb346 not found: ID does not exist"
Oct 30 19:14:55.718074 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.718045    4391 scope.go:122] "RemoveContainer" containerID="ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef"
Oct 30 19:14:55.728073 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.728057    4391 scope.go:122] "RemoveContainer" containerID="ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef"
Oct 30 19:14:55.728364 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:55.728343    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef\": container with ID starting with ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef not found: ID does not exist" containerID="ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef"
Oct 30 19:14:55.728435 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.728385    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef"} err="failed to get container status \"ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef\": rpc error: code = NotFound desc = could not find container \"ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef\": container with ID starting with ec60403070057afe43a68b8c290b284e5aa1355828153fe6807009b6dd3475ef not found: ID does not exist"
Oct 30 19:14:55.728435 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.728409    4391 scope.go:122] "RemoveContainer" containerID="9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b"
Oct 30 19:14:55.737625 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.737598    4391 scope.go:122] "RemoveContainer" containerID="9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b"
Oct 30 19:14:55.737990 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:55.737960    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b\": container with ID starting with 9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b not found: ID does not exist" containerID="9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b"
Oct 30 19:14:55.738092 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.737994    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b"} err="failed to get container status \"9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b\": rpc error: code = NotFound desc = could not find container \"9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b\": container with ID starting with 9e868def5427cbd4e245ac2951c6e8543f84b8c7a12c9506a47b6d514ad7025b not found: ID does not exist"
Oct 30 19:14:55.835994 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.835968    4391 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:14:55.836126 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.836028    4391 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 19:14:55.867807 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.867730    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:14:55.886716 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.886675    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:55.892188 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.892157    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"]
Oct 30 19:14:55.910550 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.910523    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/premium-w6-649d16-kserve-8b469dd4d-xvl5q"]
Oct 30 19:14:55.913246 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.913226    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d77b72-1171-4501-86bb-415af15ffaad-kube-api-access-5l8kr\" (UniqueName: \"kubernetes.io/projected/02d77b72-1171-4501-86bb-415af15ffaad-kube-api-access-5l8kr\") pod \"02d77b72-1171-4501-86bb-415af15ffaad\" (UID: \"02d77b72-1171-4501-86bb-415af15ffaad\") "
Oct 30 19:14:55.915280 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:55.915255    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d77b72-1171-4501-86bb-415af15ffaad-kube-api-access-5l8kr" pod "02d77b72-1171-4501-86bb-415af15ffaad" (UID: "02d77b72-1171-4501-86bb-415af15ffaad"). InnerVolumeSpecName "kube-api-access-5l8kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:56.015378 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.015347    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l8kr\" (UniqueName: \"kubernetes.io/projected/02d77b72-1171-4501-86bb-415af15ffaad-kube-api-access-5l8kr\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:56.026979 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.026950    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"]
Oct 30 19:14:56.032656 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.032626    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-w6-649d16-kserve-57654c7947-v6xpp"]
Oct 30 19:14:56.155173 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.155137    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"]
Oct 30 19:14:56.160752 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.160722    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/simulator-w6-649d16-kserve-7fd995d99c-m6vgx"]
Oct 30 19:14:56.654438 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.654365    4391 generic.go:356] "Generic (PLEG): container finished" podID="02d77b72-1171-4501-86bb-415af15ffaad" containerID="010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299" exitCode=137
Oct 30 19:14:56.654438 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.654414    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:14:56.654438 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.654431    4391 scope.go:122] "RemoveContainer" containerID="010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299"
Oct 30 19:14:56.664246 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.664226    4391 scope.go:122] "RemoveContainer" containerID="010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299"
Oct 30 19:14:56.664539 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:56.664517    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299\": container with ID starting with 010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299 not found: ID does not exist" containerID="010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299"
Oct 30 19:14:56.664611 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.664549    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299"} err="failed to get container status \"010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299\": rpc error: code = NotFound desc = could not find container \"010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299\": container with ID starting with 010d46eecf89bfae154a562ef3234eec6215863bc07000b0265ea994d8844299 not found: ID does not exist"
Oct 30 19:14:56.710978 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:56.710934    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8873-29562a\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="02d77b72-1171-4501-86bb-415af15ffaad" pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:14:56.840013 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.839986    4391 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:56.840280 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.840253    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:56.840936 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:56.840917    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:14:57.007338 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:57.007266    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d77b72-1171-4501-86bb-415af15ffaad" path="/var/lib/kubelet/pods/02d77b72-1171-4501-86bb-415af15ffaad/volumes"
Oct 30 19:14:57.007767 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:57.007752    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a69d375-f817-4870-951d-e6969a279191" path="/var/lib/kubelet/pods/0a69d375-f817-4870-951d-e6969a279191/volumes"
Oct 30 19:14:57.008169 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:57.008156    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" path="/var/lib/kubelet/pods/58ba6f26-346c-4cfa-8af2-5b46713e3c6e/volumes"
Oct 30 19:14:57.008645 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:14:57.008624    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" path="/var/lib/kubelet/pods/8bc53cc7-83c4-495b-8176-a15ef5ce4ecf/volumes"
Oct 30 19:14:57.034416 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:14:57.034387    4391 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8873-29562a\" is forbidden: User \"system:node:ip-10-0-128-92.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-92.us-west-2.compute.internal' and this object" podUID="02d77b72-1171-4501-86bb-415af15ffaad" pod="opendatahub/test-curl-8873-29562a"
Oct 30 19:15:00.204029 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.203996    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"]
Oct 30 19:15:00.204877 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.204821    4391 cpu_manager.go:414] "containerMap: removing container" podUID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" containerName="main"
Oct 30 19:15:00.204877 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.204846    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" containerName="main"
Oct 30 19:15:00.205115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.204919    4391 cpu_manager.go:414] "containerMap: removing container" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main"
Oct 30 19:15:00.205115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.204927    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main"
Oct 30 19:15:00.205115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.204950    4391 cpu_manager.go:414] "containerMap: removing container" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main"
Oct 30 19:15:00.205115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.204959    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main"
Oct 30 19:15:00.205115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.204970    4391 cpu_manager.go:414] "containerMap: removing container" podUID="02d77b72-1171-4501-86bb-415af15ffaad" containerName="test-curl-8873-29562a"
Oct 30 19:15:00.205115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.204976    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d77b72-1171-4501-86bb-415af15ffaad" containerName="test-curl-8873-29562a"
Oct 30 19:15:00.205115 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.205106    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d77b72-1171-4501-86bb-415af15ffaad" containerName="test-curl-8873-29562a"
Oct 30 19:15:00.205469 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.205123    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ba6f26-346c-4cfa-8af2-5b46713e3c6e" containerName="main"
Oct 30 19:15:00.205469 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.205144    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc53cc7-83c4-495b-8176-a15ef5ce4ecf" containerName="main"
Oct 30 19:15:00.205469 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.205158    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a69d375-f817-4870-951d-e6969a279191" containerName="main"
Oct 30 19:15:00.210162 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.210140    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"]
Oct 30 19:15:00.210277 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.210258    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"
Oct 30 19:15:00.214090 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.214071    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"
Oct 30 19:15:00.239965 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.239933    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"]
Oct 30 19:15:00.247620 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.245836    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"
Oct 30 19:15:00.263457 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.263433    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"]
Oct 30 19:15:00.264045 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.264019    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cnhd\" (UniqueName: \"kubernetes.io/projected/84a719e5-b8cd-421a-9869-83b39f95391f-kube-api-access-9cnhd\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd\" (UID: \"84a719e5-b8cd-421a-9869-83b39f95391f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"
Oct 30 19:15:00.264163 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.264068    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpzcz\" (UniqueName: \"kubernetes.io/projected/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-kube-api-access-mpzcz\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w\" (UID: \"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"
Oct 30 19:15:00.264279 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.264248    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a719e5-b8cd-421a-9869-83b39f95391f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd\" (UID: \"84a719e5-b8cd-421a-9869-83b39f95391f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"
Oct 30 19:15:00.264406 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.264312    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w\" (UID: \"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"
Oct 30 19:15:00.267849 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.267833    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"
Oct 30 19:15:00.305389 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.305353    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"]
Oct 30 19:15:00.332015 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.331972    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:15:00.357668 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.357634    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"]
Oct 30 19:15:00.359549 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.359520    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"]
Oct 30 19:15:00.365472 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.365449    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w\" (UID: \"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"
Oct 30 19:15:00.365603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.365563    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca127f30-3020-42f7-af2d-d23ca6c79fcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6\" (UID: \"ca127f30-3020-42f7-af2d-d23ca6c79fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"
Oct 30 19:15:00.365674 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.365625    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7q68\" (UniqueName: \"kubernetes.io/projected/6f03dc4b-361e-419b-a1ae-46a2146e0e98-kube-api-access-w7q68\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz\" (UID: \"6f03dc4b-361e-419b-a1ae-46a2146e0e98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"
Oct 30 19:15:00.365745 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.365729    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cnhd\" (UniqueName: \"kubernetes.io/projected/84a719e5-b8cd-421a-9869-83b39f95391f-kube-api-access-9cnhd\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd\" (UID: \"84a719e5-b8cd-421a-9869-83b39f95391f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"
Oct 30 19:15:00.365811 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.365752    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f03dc4b-361e-419b-a1ae-46a2146e0e98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz\" (UID: \"6f03dc4b-361e-419b-a1ae-46a2146e0e98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"
Oct 30 19:15:00.365811 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.365774    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpzcz\" (UniqueName: \"kubernetes.io/projected/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-kube-api-access-mpzcz\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w\" (UID: \"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"
Oct 30 19:15:00.365929 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.365884    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a719e5-b8cd-421a-9869-83b39f95391f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd\" (UID: \"84a719e5-b8cd-421a-9869-83b39f95391f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"
Oct 30 19:15:00.365983 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.365926    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7sk2\" (UniqueName: \"kubernetes.io/projected/ca127f30-3020-42f7-af2d-d23ca6c79fcb-kube-api-access-w7sk2\") pod \"maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6\" (UID: \"ca127f30-3020-42f7-af2d-d23ca6c79fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"
Oct 30 19:15:00.366293 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.366259    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w\" (UID: \"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"
Oct 30 19:15:00.366820 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.366792    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a719e5-b8cd-421a-9869-83b39f95391f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd\" (UID: \"84a719e5-b8cd-421a-9869-83b39f95391f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"
Oct 30 19:15:00.367426 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.367409    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"]
Oct 30 19:15:00.420988 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.420957    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cnhd\" (UniqueName: \"kubernetes.io/projected/84a719e5-b8cd-421a-9869-83b39f95391f-kube-api-access-9cnhd\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd\" (UID: \"84a719e5-b8cd-421a-9869-83b39f95391f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"
Oct 30 19:15:00.421219 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.421198    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpzcz\" (UniqueName: \"kubernetes.io/projected/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-kube-api-access-mpzcz\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w\" (UID: \"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"
Oct 30 19:15:00.466845 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.466768    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7sk2\" (UniqueName: \"kubernetes.io/projected/ca127f30-3020-42f7-af2d-d23ca6c79fcb-kube-api-access-w7sk2\") pod \"maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6\" (UID: \"ca127f30-3020-42f7-af2d-d23ca6c79fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"
Oct 30 19:15:00.466969 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.466904    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca127f30-3020-42f7-af2d-d23ca6c79fcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6\" (UID: \"ca127f30-3020-42f7-af2d-d23ca6c79fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"
Oct 30 19:15:00.466969 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.466938    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7q68\" (UniqueName: \"kubernetes.io/projected/6f03dc4b-361e-419b-a1ae-46a2146e0e98-kube-api-access-w7q68\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz\" (UID: \"6f03dc4b-361e-419b-a1ae-46a2146e0e98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"
Oct 30 19:15:00.467067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.466996    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f03dc4b-361e-419b-a1ae-46a2146e0e98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz\" (UID: \"6f03dc4b-361e-419b-a1ae-46a2146e0e98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"
Oct 30 19:15:00.467568 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.467546    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca127f30-3020-42f7-af2d-d23ca6c79fcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6\" (UID: \"ca127f30-3020-42f7-af2d-d23ca6c79fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"
Oct 30 19:15:00.467687 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.467667    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f03dc4b-361e-419b-a1ae-46a2146e0e98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz\" (UID: \"6f03dc4b-361e-419b-a1ae-46a2146e0e98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"
Oct 30 19:15:00.523072 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.523047    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7sk2\" (UniqueName: \"kubernetes.io/projected/ca127f30-3020-42f7-af2d-d23ca6c79fcb-kube-api-access-w7sk2\") pod \"maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6\" (UID: \"ca127f30-3020-42f7-af2d-d23ca6c79fcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"
Oct 30 19:15:00.523808 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.523786    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7q68\" (UniqueName: \"kubernetes.io/projected/6f03dc4b-361e-419b-a1ae-46a2146e0e98-kube-api-access-w7q68\") pod \"maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz\" (UID: \"6f03dc4b-361e-419b-a1ae-46a2146e0e98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"
Oct 30 19:15:00.530757 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.530732    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"
Oct 30 19:15:00.542946 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.542918    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"
Oct 30 19:15:00.559826 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.559804    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"
Oct 30 19:15:00.581157 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.580967    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"
Oct 30 19:15:00.913726 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.913689    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"]
Oct 30 19:15:00.913914 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:15:00.913884    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a719e5_b8cd_421a_9869_83b39f95391f.slice/crio-f0a2824e31885d804d783fc798e7338d3be40c15490b80cea6091f51c827cb66 WatchSource:0}: Error finding container f0a2824e31885d804d783fc798e7338d3be40c15490b80cea6091f51c827cb66: Status 404 returned error can't find the container with id f0a2824e31885d804d783fc798e7338d3be40c15490b80cea6091f51c827cb66
Oct 30 19:15:00.956458 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.956430    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"]
Oct 30 19:15:00.956834 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:15:00.956810    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc3da2e_6fb6_4e35_9777_a57f5ccb0019.slice/crio-cd4091fe94279d8535042ab8b110c6a4c137a0aafd4e67f1d797a2dba9d326bb WatchSource:0}: Error finding container cd4091fe94279d8535042ab8b110c6a4c137a0aafd4e67f1d797a2dba9d326bb: Status 404 returned error can't find the container with id cd4091fe94279d8535042ab8b110c6a4c137a0aafd4e67f1d797a2dba9d326bb
Oct 30 19:15:00.960812 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:00.960788    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"]
Oct 30 19:15:00.961251 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:15:00.961229    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f03dc4b_361e_419b_a1ae_46a2146e0e98.slice/crio-a31f7fb392a68895703cff4df6147e56235d82459e13810a47f06fdef2c04427 WatchSource:0}: Error finding container a31f7fb392a68895703cff4df6147e56235d82459e13810a47f06fdef2c04427: Status 404 returned error can't find the container with id a31f7fb392a68895703cff4df6147e56235d82459e13810a47f06fdef2c04427
Oct 30 19:15:01.009753 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:15:01.009725    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca127f30_3020_42f7_af2d_d23ca6c79fcb.slice/crio-60f06b0109772a4da332994fbaec0ec97cd9c470c9c103368d279b4846369402 WatchSource:0}: Error finding container 60f06b0109772a4da332994fbaec0ec97cd9c470c9c103368d279b4846369402: Status 404 returned error can't find the container with id 60f06b0109772a4da332994fbaec0ec97cd9c470c9c103368d279b4846369402
Oct 30 19:15:01.014292 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.014266    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"]
Oct 30 19:15:01.697482 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.697443    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" event={"ID":"6f03dc4b-361e-419b-a1ae-46a2146e0e98","Type":"ContainerStarted","Data":"4e21d16d7604ebc0f7a9f4bd411717fb7f154c91f9e9b17eeb5a15e709506883"}
Oct 30 19:15:01.698069 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.697503    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" event={"ID":"6f03dc4b-361e-419b-a1ae-46a2146e0e98","Type":"ContainerStarted","Data":"a31f7fb392a68895703cff4df6147e56235d82459e13810a47f06fdef2c04427"}
Oct 30 19:15:01.699166 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.699129    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" event={"ID":"84a719e5-b8cd-421a-9869-83b39f95391f","Type":"ContainerStarted","Data":"5208b48a6b38fe9dbf3aa4e49e3bbf2e22d7cb432e3829992be4352fc1b8c32b"}
Oct 30 19:15:01.699296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.699180    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" event={"ID":"84a719e5-b8cd-421a-9869-83b39f95391f","Type":"ContainerStarted","Data":"f0a2824e31885d804d783fc798e7338d3be40c15490b80cea6091f51c827cb66"}
Oct 30 19:15:01.700889 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.700863    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" event={"ID":"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019","Type":"ContainerStarted","Data":"c5c03d307be337868eff1e14ca4de92e21c2f292aa8d66cc946f45a59c9342c3"}
Oct 30 19:15:01.700985 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.700912    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" event={"ID":"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019","Type":"ContainerStarted","Data":"cd4091fe94279d8535042ab8b110c6a4c137a0aafd4e67f1d797a2dba9d326bb"}
Oct 30 19:15:01.703909 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.703881    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" event={"ID":"ca127f30-3020-42f7-af2d-d23ca6c79fcb","Type":"ContainerStarted","Data":"9f2609a00d4e0963137f12274ac230179634b9be2e42fe3dbbb442b36b40e09e"}
Oct 30 19:15:01.704002 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.703939    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" event={"ID":"ca127f30-3020-42f7-af2d-d23ca6c79fcb","Type":"ContainerStarted","Data":"60f06b0109772a4da332994fbaec0ec97cd9c470c9c103368d279b4846369402"}
Oct 30 19:15:01.805844 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.805782    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" podStartSLOduration=1.805767605 podStartE2EDuration="1.805767605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:01.804959804 +0000 UTC m=+2296.915693068" watchObservedRunningTime="2026-10-30 19:15:01.805767605 +0000 UTC m=+2296.916500889"
Oct 30 19:15:01.917458 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:01.917396    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" podStartSLOduration=1.917381303 podStartE2EDuration="1.917381303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:01.916118666 +0000 UTC m=+2297.026851950" watchObservedRunningTime="2026-10-30 19:15:01.917381303 +0000 UTC m=+2297.028114587"
Oct 30 19:15:02.028632 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.028485    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" podStartSLOduration=2.028466203 podStartE2EDuration="2.028466203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.027226446 +0000 UTC m=+2297.137959733" watchObservedRunningTime="2026-10-30 19:15:02.028466203 +0000 UTC m=+2297.139199487"
Oct 30 19:15:02.182746 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.182665    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" podStartSLOduration=2.182639056 podStartE2EDuration="2.182639056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.145456086 +0000 UTC m=+2297.256189361" watchObservedRunningTime="2026-10-30 19:15:02.182639056 +0000 UTC m=+2297.293372341"
Oct 30 19:15:02.712507 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.712469    4391 generic.go:356] "Generic (PLEG): container finished" podID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerID="9f2609a00d4e0963137f12274ac230179634b9be2e42fe3dbbb442b36b40e09e" exitCode=60
Oct 30 19:15:02.713006 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.712553    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" event={"ID":"ca127f30-3020-42f7-af2d-d23ca6c79fcb","Type":"ContainerDied","Data":"9f2609a00d4e0963137f12274ac230179634b9be2e42fe3dbbb442b36b40e09e"}
Oct 30 19:15:02.713006 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.712835    4391 scope.go:122] "RemoveContainer" containerID="9f2609a00d4e0963137f12274ac230179634b9be2e42fe3dbbb442b36b40e09e"
Oct 30 19:15:02.714336 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.714314    4391 generic.go:356] "Generic (PLEG): container finished" podID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerID="4e21d16d7604ebc0f7a9f4bd411717fb7f154c91f9e9b17eeb5a15e709506883" exitCode=60
Oct 30 19:15:02.714432 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.714421    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" event={"ID":"6f03dc4b-361e-419b-a1ae-46a2146e0e98","Type":"ContainerDied","Data":"4e21d16d7604ebc0f7a9f4bd411717fb7f154c91f9e9b17eeb5a15e709506883"}
Oct 30 19:15:02.714686 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.714661    4391 scope.go:122] "RemoveContainer" containerID="4e21d16d7604ebc0f7a9f4bd411717fb7f154c91f9e9b17eeb5a15e709506883"
Oct 30 19:15:02.716248 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.716179    4391 generic.go:356] "Generic (PLEG): container finished" podID="84a719e5-b8cd-421a-9869-83b39f95391f" containerID="5208b48a6b38fe9dbf3aa4e49e3bbf2e22d7cb432e3829992be4352fc1b8c32b" exitCode=60
Oct 30 19:15:02.716303 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.716257    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" event={"ID":"84a719e5-b8cd-421a-9869-83b39f95391f","Type":"ContainerDied","Data":"5208b48a6b38fe9dbf3aa4e49e3bbf2e22d7cb432e3829992be4352fc1b8c32b"}
Oct 30 19:15:02.716448 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.716434    4391 scope.go:122] "RemoveContainer" containerID="5208b48a6b38fe9dbf3aa4e49e3bbf2e22d7cb432e3829992be4352fc1b8c32b"
Oct 30 19:15:02.718048 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.718027    4391 generic.go:356] "Generic (PLEG): container finished" podID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerID="c5c03d307be337868eff1e14ca4de92e21c2f292aa8d66cc946f45a59c9342c3" exitCode=60
Oct 30 19:15:02.718136 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.718086    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" event={"ID":"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019","Type":"ContainerDied","Data":"c5c03d307be337868eff1e14ca4de92e21c2f292aa8d66cc946f45a59c9342c3"}
Oct 30 19:15:02.718352 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:02.718334    4391 scope.go:122] "RemoveContainer" containerID="c5c03d307be337868eff1e14ca4de92e21c2f292aa8d66cc946f45a59c9342c3"
Oct 30 19:15:03.726656 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:03.726618    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" event={"ID":"ca127f30-3020-42f7-af2d-d23ca6c79fcb","Type":"ContainerStarted","Data":"41cf4d32b90d497eebed379be71f8a01ff0a41cf73223839455e7c906e6f4f54"}
Oct 30 19:15:03.728461 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:03.728430    4391 generic.go:356] "Generic (PLEG): container finished" podID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerID="e96987e7f4e0d30d2fec323880fd85f066481bdce4f6731b30a6af61f4e52df0" exitCode=60
Oct 30 19:15:03.728628 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:03.728496    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" event={"ID":"6f03dc4b-361e-419b-a1ae-46a2146e0e98","Type":"ContainerDied","Data":"e96987e7f4e0d30d2fec323880fd85f066481bdce4f6731b30a6af61f4e52df0"}
Oct 30 19:15:03.728628 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:03.728551    4391 scope.go:122] "RemoveContainer" containerID="4e21d16d7604ebc0f7a9f4bd411717fb7f154c91f9e9b17eeb5a15e709506883"
Oct 30 19:15:03.729005 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:03.728987    4391 scope.go:122] "RemoveContainer" containerID="e96987e7f4e0d30d2fec323880fd85f066481bdce4f6731b30a6af61f4e52df0"
Oct 30 19:15:03.729308 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:03.729279    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz_odh-ai-gateway-infra(6f03dc4b-361e-419b-a1ae-46a2146e0e98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98"
Oct 30 19:15:03.730645 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:03.730622    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" event={"ID":"84a719e5-b8cd-421a-9869-83b39f95391f","Type":"ContainerStarted","Data":"3c08d5590b054cea44c82f06176b72c8448b3a42d74359eb1fe4fb0a3884c53c"}
Oct 30 19:15:03.732607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:03.732559    4391 generic.go:356] "Generic (PLEG): container finished" podID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerID="d3d03caa765b973246ae14c2cdfa4c096f735a5867b865eb0c7c7b81b06feee8" exitCode=60
Oct 30 19:15:03.732684 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:03.732624    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" event={"ID":"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019","Type":"ContainerDied","Data":"d3d03caa765b973246ae14c2cdfa4c096f735a5867b865eb0c7c7b81b06feee8"}
Oct 30 19:15:03.732844 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:03.732828    4391 scope.go:122] "RemoveContainer" containerID="d3d03caa765b973246ae14c2cdfa4c096f735a5867b865eb0c7c7b81b06feee8"
Oct 30 19:15:03.733064 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:03.733047    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w_odh-ai-gateway-infra(1bc3da2e-6fb6-4e35-9777-a57f5ccb0019)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019"
Oct 30 19:15:03.856214 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:03.856183    4391 scope.go:122] "RemoveContainer" containerID="c5c03d307be337868eff1e14ca4de92e21c2f292aa8d66cc946f45a59c9342c3"
Oct 30 19:15:04.742296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:04.742256    4391 generic.go:356] "Generic (PLEG): container finished" podID="84a719e5-b8cd-421a-9869-83b39f95391f" containerID="3c08d5590b054cea44c82f06176b72c8448b3a42d74359eb1fe4fb0a3884c53c" exitCode=60
Oct 30 19:15:04.742894 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:04.742334    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" event={"ID":"84a719e5-b8cd-421a-9869-83b39f95391f","Type":"ContainerDied","Data":"3c08d5590b054cea44c82f06176b72c8448b3a42d74359eb1fe4fb0a3884c53c"}
Oct 30 19:15:04.742894 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:04.742402    4391 scope.go:122] "RemoveContainer" containerID="5208b48a6b38fe9dbf3aa4e49e3bbf2e22d7cb432e3829992be4352fc1b8c32b"
Oct 30 19:15:04.742894 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:04.742718    4391 scope.go:122] "RemoveContainer" containerID="3c08d5590b054cea44c82f06176b72c8448b3a42d74359eb1fe4fb0a3884c53c"
Oct 30 19:15:04.743078 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:04.743053    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd_odh-ai-gateway-infra(84a719e5-b8cd-421a-9869-83b39f95391f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" podUID="84a719e5-b8cd-421a-9869-83b39f95391f"
Oct 30 19:15:04.744542 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:04.744527    4391 scope.go:122] "RemoveContainer" containerID="d3d03caa765b973246ae14c2cdfa4c096f735a5867b865eb0c7c7b81b06feee8"
Oct 30 19:15:04.744840 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:04.744820    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w_odh-ai-gateway-infra(1bc3da2e-6fb6-4e35-9777-a57f5ccb0019)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019"
Oct 30 19:15:04.746417 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:04.746391    4391 generic.go:356] "Generic (PLEG): container finished" podID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerID="41cf4d32b90d497eebed379be71f8a01ff0a41cf73223839455e7c906e6f4f54" exitCode=60
Oct 30 19:15:04.746537 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:04.746462    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" event={"ID":"ca127f30-3020-42f7-af2d-d23ca6c79fcb","Type":"ContainerDied","Data":"41cf4d32b90d497eebed379be71f8a01ff0a41cf73223839455e7c906e6f4f54"}
Oct 30 19:15:04.746787 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:04.746760    4391 scope.go:122] "RemoveContainer" containerID="41cf4d32b90d497eebed379be71f8a01ff0a41cf73223839455e7c906e6f4f54"
Oct 30 19:15:04.747037 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:04.747013    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6_odh-ai-gateway-infra(ca127f30-3020-42f7-af2d-d23ca6c79fcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb"
Oct 30 19:15:04.749104 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:04.749085    4391 scope.go:122] "RemoveContainer" containerID="e96987e7f4e0d30d2fec323880fd85f066481bdce4f6731b30a6af61f4e52df0"
Oct 30 19:15:04.749314 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:04.749293    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz_odh-ai-gateway-infra(6f03dc4b-361e-419b-a1ae-46a2146e0e98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98"
Oct 30 19:15:04.757794 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:04.757768    4391 scope.go:122] "RemoveContainer" containerID="9f2609a00d4e0963137f12274ac230179634b9be2e42fe3dbbb442b36b40e09e"
Oct 30 19:15:05.758212 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:05.758173    4391 scope.go:122] "RemoveContainer" containerID="41cf4d32b90d497eebed379be71f8a01ff0a41cf73223839455e7c906e6f4f54"
Oct 30 19:15:05.758732 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:05.758559    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6_odh-ai-gateway-infra(ca127f30-3020-42f7-af2d-d23ca6c79fcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb"
Oct 30 19:15:05.759908 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:05.759886    4391 scope.go:122] "RemoveContainer" containerID="3c08d5590b054cea44c82f06176b72c8448b3a42d74359eb1fe4fb0a3884c53c"
Oct 30 19:15:05.760180 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:05.760158    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd_odh-ai-gateway-infra(84a719e5-b8cd-421a-9869-83b39f95391f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" podUID="84a719e5-b8cd-421a-9869-83b39f95391f"
Oct 30 19:15:08.489561 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.489519    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"]
Oct 30 19:15:08.490058 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.489794    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8" podUID="71279088-1baf-4ac7-a020-a22ce6f5f76b" containerName="maas-api" containerID="cri-o://8726739591f36f4551c869beb7ff04d4af8d39e4d1a9193db97c9eac2601353c" gracePeriod=30
Oct 30 19:15:08.841317 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.841280    4391 generic.go:356] "Generic (PLEG): container finished" podID="71279088-1baf-4ac7-a020-a22ce6f5f76b" containerID="8726739591f36f4551c869beb7ff04d4af8d39e4d1a9193db97c9eac2601353c" exitCode=0
Oct 30 19:15:08.841628 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.841367    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8" event={"ID":"71279088-1baf-4ac7-a020-a22ce6f5f76b","Type":"ContainerDied","Data":"8726739591f36f4551c869beb7ff04d4af8d39e4d1a9193db97c9eac2601353c"}
Oct 30 19:15:08.879603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.868122    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:15:08.949142 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.948129    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71279088-1baf-4ac7-a020-a22ce6f5f76b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71279088-1baf-4ac7-a020-a22ce6f5f76b-openshift-service-ca\") pod \"71279088-1baf-4ac7-a020-a22ce6f5f76b\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") "
Oct 30 19:15:08.949142 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.948270    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71279088-1baf-4ac7-a020-a22ce6f5f76b-kube-api-access-txspj\" (UniqueName: \"kubernetes.io/projected/71279088-1baf-4ac7-a020-a22ce6f5f76b-kube-api-access-txspj\") pod \"71279088-1baf-4ac7-a020-a22ce6f5f76b\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") "
Oct 30 19:15:08.949142 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.948305    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-metrics-certs\") pod \"71279088-1baf-4ac7-a020-a22ce6f5f76b\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") "
Oct 30 19:15:08.949142 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.948391    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-maas-api-tls\") pod \"71279088-1baf-4ac7-a020-a22ce6f5f76b\" (UID: \"71279088-1baf-4ac7-a020-a22ce6f5f76b\") "
Oct 30 19:15:08.952880 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.952827    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71279088-1baf-4ac7-a020-a22ce6f5f76b-kube-api-access-txspj" pod "71279088-1baf-4ac7-a020-a22ce6f5f76b" (UID: "71279088-1baf-4ac7-a020-a22ce6f5f76b"). InnerVolumeSpecName "kube-api-access-txspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:08.953259 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.953234    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-maas-api-tls" pod "71279088-1baf-4ac7-a020-a22ce6f5f76b" (UID: "71279088-1baf-4ac7-a020-a22ce6f5f76b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:08.955221 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.955185    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-metrics-certs" pod "71279088-1baf-4ac7-a020-a22ce6f5f76b" (UID: "71279088-1baf-4ac7-a020-a22ce6f5f76b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:08.961812 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:08.961784    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71279088-1baf-4ac7-a020-a22ce6f5f76b-openshift-service-ca" pod "71279088-1baf-4ac7-a020-a22ce6f5f76b" (UID: "71279088-1baf-4ac7-a020-a22ce6f5f76b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:09.051062 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:09.051022    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txspj\" (UniqueName: \"kubernetes.io/projected/71279088-1baf-4ac7-a020-a22ce6f5f76b-kube-api-access-txspj\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:09.051062 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:09.051065    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:09.051302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:09.051080    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71279088-1baf-4ac7-a020-a22ce6f5f76b-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:09.051302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:09.051094    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71279088-1baf-4ac7-a020-a22ce6f5f76b-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:09.850441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:09.850412    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"
Oct 30 19:15:09.850925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:09.850455    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8" event={"ID":"71279088-1baf-4ac7-a020-a22ce6f5f76b","Type":"ContainerDied","Data":"84e15590da5d34d67c041ee295563d8236ddf65750e4f79b1acec174c5dbc480"}
Oct 30 19:15:09.850925 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:09.850519    4391 scope.go:122] "RemoveContainer" containerID="8726739591f36f4551c869beb7ff04d4af8d39e4d1a9193db97c9eac2601353c"
Oct 30 19:15:10.008111 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:10.008078    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"]
Oct 30 19:15:10.054228 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:10.054191    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-05565d-bf88f6449-sq8x8"]
Oct 30 19:15:11.006756 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:11.006718    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71279088-1baf-4ac7-a020-a22ce6f5f76b" path="/var/lib/kubelet/pods/71279088-1baf-4ac7-a020-a22ce6f5f76b/volumes"
Oct 30 19:15:11.253214 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:11.253173    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:15:15.003039 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:15.003006    4391 scope.go:122] "RemoveContainer" containerID="d3d03caa765b973246ae14c2cdfa4c096f735a5867b865eb0c7c7b81b06feee8"
Oct 30 19:15:15.003449 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:15.003143    4391 scope.go:122] "RemoveContainer" containerID="41cf4d32b90d497eebed379be71f8a01ff0a41cf73223839455e7c906e6f4f54"
Oct 30 19:15:15.905265 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:15.905229    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" event={"ID":"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019","Type":"ContainerStarted","Data":"455832bb36407994b2713cf6a373d032c3ead1fdfe53a1abb187b832c6229508"}
Oct 30 19:15:15.907362 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:15.907333    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" event={"ID":"ca127f30-3020-42f7-af2d-d23ca6c79fcb","Type":"ContainerStarted","Data":"5784600304d424795499ee5679474924520df9b3a2e4b09e300b1d843b039794"}
Oct 30 19:15:16.002831 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.002304    4391 scope.go:122] "RemoveContainer" containerID="3c08d5590b054cea44c82f06176b72c8448b3a42d74359eb1fe4fb0a3884c53c"
Oct 30 19:15:16.093019 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.090626    4391 scope.go:122] "RemoveContainer" containerID="e96987e7f4e0d30d2fec323880fd85f066481bdce4f6731b30a6af61f4e52df0"
Oct 30 19:15:16.917743 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.917711    4391 generic.go:356] "Generic (PLEG): container finished" podID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerID="5784600304d424795499ee5679474924520df9b3a2e4b09e300b1d843b039794" exitCode=60
Oct 30 19:15:16.917964 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.917792    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" event={"ID":"ca127f30-3020-42f7-af2d-d23ca6c79fcb","Type":"ContainerDied","Data":"5784600304d424795499ee5679474924520df9b3a2e4b09e300b1d843b039794"}
Oct 30 19:15:16.917964 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.917848    4391 scope.go:122] "RemoveContainer" containerID="41cf4d32b90d497eebed379be71f8a01ff0a41cf73223839455e7c906e6f4f54"
Oct 30 19:15:16.918310 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.918291    4391 scope.go:122] "RemoveContainer" containerID="5784600304d424795499ee5679474924520df9b3a2e4b09e300b1d843b039794"
Oct 30 19:15:16.918700 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:16.918672    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6_odh-ai-gateway-infra(ca127f30-3020-42f7-af2d-d23ca6c79fcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb"
Oct 30 19:15:16.920270 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.920243    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" event={"ID":"6f03dc4b-361e-419b-a1ae-46a2146e0e98","Type":"ContainerStarted","Data":"c014d188cc1b07467fa5b35da9d16e65ae71ef3e27a9264582577806f01e3b91"}
Oct 30 19:15:16.922479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.922456    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" event={"ID":"84a719e5-b8cd-421a-9869-83b39f95391f","Type":"ContainerStarted","Data":"9bea4fea90f3045c5319109c70386c216a2777ee21a8be4fb7d3588b754391fb"}
Oct 30 19:15:16.924467 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.924444    4391 generic.go:356] "Generic (PLEG): container finished" podID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerID="455832bb36407994b2713cf6a373d032c3ead1fdfe53a1abb187b832c6229508" exitCode=60
Oct 30 19:15:16.924565 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.924519    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" event={"ID":"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019","Type":"ContainerDied","Data":"455832bb36407994b2713cf6a373d032c3ead1fdfe53a1abb187b832c6229508"}
Oct 30 19:15:16.924941 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.924923    4391 scope.go:122] "RemoveContainer" containerID="455832bb36407994b2713cf6a373d032c3ead1fdfe53a1abb187b832c6229508"
Oct 30 19:15:16.925229 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:16.925207    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w_odh-ai-gateway-infra(1bc3da2e-6fb6-4e35-9777-a57f5ccb0019)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019"
Oct 30 19:15:16.932461 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:16.932439    4391 scope.go:122] "RemoveContainer" containerID="d3d03caa765b973246ae14c2cdfa4c096f735a5867b865eb0c7c7b81b06feee8"
Oct 30 19:15:17.029926 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.029887    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"]
Oct 30 19:15:17.203247 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.203215    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"]
Oct 30 19:15:17.516145 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.516043    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"]
Oct 30 19:15:17.933628 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.933562    4391 generic.go:356] "Generic (PLEG): container finished" podID="84a719e5-b8cd-421a-9869-83b39f95391f" containerID="9bea4fea90f3045c5319109c70386c216a2777ee21a8be4fb7d3588b754391fb" exitCode=60
Oct 30 19:15:17.933628 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.933609    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" event={"ID":"84a719e5-b8cd-421a-9869-83b39f95391f","Type":"ContainerDied","Data":"9bea4fea90f3045c5319109c70386c216a2777ee21a8be4fb7d3588b754391fb"}
Oct 30 19:15:17.933855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.933671    4391 scope.go:122] "RemoveContainer" containerID="3c08d5590b054cea44c82f06176b72c8448b3a42d74359eb1fe4fb0a3884c53c"
Oct 30 19:15:17.934067 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.934040    4391 scope.go:122] "RemoveContainer" containerID="9bea4fea90f3045c5319109c70386c216a2777ee21a8be4fb7d3588b754391fb"
Oct 30 19:15:17.934412 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:17.934390    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd_odh-ai-gateway-infra(84a719e5-b8cd-421a-9869-83b39f95391f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" podUID="84a719e5-b8cd-421a-9869-83b39f95391f"
Oct 30 19:15:17.940078 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.939929    4391 generic.go:356] "Generic (PLEG): container finished" podID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerID="c014d188cc1b07467fa5b35da9d16e65ae71ef3e27a9264582577806f01e3b91" exitCode=60
Oct 30 19:15:17.940078 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.940005    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" event={"ID":"6f03dc4b-361e-419b-a1ae-46a2146e0e98","Type":"ContainerDied","Data":"c014d188cc1b07467fa5b35da9d16e65ae71ef3e27a9264582577806f01e3b91"}
Oct 30 19:15:17.940422 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.940402    4391 scope.go:122] "RemoveContainer" containerID="c014d188cc1b07467fa5b35da9d16e65ae71ef3e27a9264582577806f01e3b91"
Oct 30 19:15:17.940819 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:17.940775    4391 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz_odh-ai-gateway-infra(6f03dc4b-361e-419b-a1ae-46a2146e0e98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98"
Oct 30 19:15:17.947470 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:17.947438    4391 scope.go:122] "RemoveContainer" containerID="e96987e7f4e0d30d2fec323880fd85f066481bdce4f6731b30a6af61f4e52df0"
Oct 30 19:15:18.103223 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.103191    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"
Oct 30 19:15:18.131454 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.131434    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"
Oct 30 19:15:18.161312 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.161279    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"]
Oct 30 19:15:18.197233 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.197142    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-maas-api-service-ca\") pod \"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019\" (UID: \"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019\") "
Oct 30 19:15:18.197233 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.197228    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-kube-api-access-mpzcz\" (UniqueName: \"kubernetes.io/projected/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-kube-api-access-mpzcz\") pod \"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019\" (UID: \"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019\") "
Oct 30 19:15:18.197442 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.197333    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca127f30-3020-42f7-af2d-d23ca6c79fcb-kube-api-access-w7sk2\" (UniqueName: \"kubernetes.io/projected/ca127f30-3020-42f7-af2d-d23ca6c79fcb-kube-api-access-w7sk2\") pod \"ca127f30-3020-42f7-af2d-d23ca6c79fcb\" (UID: \"ca127f30-3020-42f7-af2d-d23ca6c79fcb\") "
Oct 30 19:15:18.197612 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.197574    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-maas-api-service-ca" pod "1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" (UID: "1bc3da2e-6fb6-4e35-9777-a57f5ccb0019"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:18.197958 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.197937    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-maas-api-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.199529 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.199502    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca127f30-3020-42f7-af2d-d23ca6c79fcb-kube-api-access-w7sk2" pod "ca127f30-3020-42f7-af2d-d23ca6c79fcb" (UID: "ca127f30-3020-42f7-af2d-d23ca6c79fcb"). InnerVolumeSpecName "kube-api-access-w7sk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:18.199671 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.199652    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-kube-api-access-mpzcz" pod "1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" (UID: "1bc3da2e-6fb6-4e35-9777-a57f5ccb0019"). InnerVolumeSpecName "kube-api-access-mpzcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:18.270219 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.270181    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"]
Oct 30 19:15:18.298724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.298695    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca127f30-3020-42f7-af2d-d23ca6c79fcb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca127f30-3020-42f7-af2d-d23ca6c79fcb-maas-api-service-ca\") pod \"ca127f30-3020-42f7-af2d-d23ca6c79fcb\" (UID: \"ca127f30-3020-42f7-af2d-d23ca6c79fcb\") "
Oct 30 19:15:18.299118 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.299095    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca127f30-3020-42f7-af2d-d23ca6c79fcb-maas-api-service-ca" pod "ca127f30-3020-42f7-af2d-d23ca6c79fcb" (UID: "ca127f30-3020-42f7-af2d-d23ca6c79fcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:18.299324 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.299309    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca127f30-3020-42f7-af2d-d23ca6c79fcb-maas-api-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.299361 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.299328    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpzcz\" (UniqueName: \"kubernetes.io/projected/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019-kube-api-access-mpzcz\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.299361 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.299338    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7sk2\" (UniqueName: \"kubernetes.io/projected/ca127f30-3020-42f7-af2d-d23ca6c79fcb-kube-api-access-w7sk2\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.949990 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.949956    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"
Oct 30 19:15:18.950209 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.949980    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6" event={"ID":"ca127f30-3020-42f7-af2d-d23ca6c79fcb","Type":"ContainerDied","Data":"60f06b0109772a4da332994fbaec0ec97cd9c470c9c103368d279b4846369402"}
Oct 30 19:15:18.950209 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.950061    4391 scope.go:122] "RemoveContainer" containerID="5784600304d424795499ee5679474924520df9b3a2e4b09e300b1d843b039794"
Oct 30 19:15:18.955651 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.955623    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"
Oct 30 19:15:18.955651 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.955632    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w" event={"ID":"1bc3da2e-6fb6-4e35-9777-a57f5ccb0019","Type":"ContainerDied","Data":"cd4091fe94279d8535042ab8b110c6a4c137a0aafd4e67f1d797a2dba9d326bb"}
Oct 30 19:15:18.970082 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:18.970057    4391 scope.go:122] "RemoveContainer" containerID="455832bb36407994b2713cf6a373d032c3ead1fdfe53a1abb187b832c6229508"
Oct 30 19:15:19.116072 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.116039    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"]
Oct 30 19:15:19.125831 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.125402    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8e31a7-29889795-9cgd6"]
Oct 30 19:15:19.128494 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.128467    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"
Oct 30 19:15:19.146231 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.146207    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"
Oct 30 19:15:19.210445 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.210359    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a719e5-b8cd-421a-9869-83b39f95391f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a719e5-b8cd-421a-9869-83b39f95391f-maas-api-service-ca\") pod \"84a719e5-b8cd-421a-9869-83b39f95391f\" (UID: \"84a719e5-b8cd-421a-9869-83b39f95391f\") "
Oct 30 19:15:19.210445 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.210433    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a719e5-b8cd-421a-9869-83b39f95391f-kube-api-access-9cnhd\" (UniqueName: \"kubernetes.io/projected/84a719e5-b8cd-421a-9869-83b39f95391f-kube-api-access-9cnhd\") pod \"84a719e5-b8cd-421a-9869-83b39f95391f\" (UID: \"84a719e5-b8cd-421a-9869-83b39f95391f\") "
Oct 30 19:15:19.210719 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.210500    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f03dc4b-361e-419b-a1ae-46a2146e0e98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f03dc4b-361e-419b-a1ae-46a2146e0e98-maas-api-service-ca\") pod \"6f03dc4b-361e-419b-a1ae-46a2146e0e98\" (UID: \"6f03dc4b-361e-419b-a1ae-46a2146e0e98\") "
Oct 30 19:15:19.210719 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.210556    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f03dc4b-361e-419b-a1ae-46a2146e0e98-kube-api-access-w7q68\" (UniqueName: \"kubernetes.io/projected/6f03dc4b-361e-419b-a1ae-46a2146e0e98-kube-api-access-w7q68\") pod \"6f03dc4b-361e-419b-a1ae-46a2146e0e98\" (UID: \"6f03dc4b-361e-419b-a1ae-46a2146e0e98\") "
Oct 30 19:15:19.210877 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.210853    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f03dc4b-361e-419b-a1ae-46a2146e0e98-maas-api-service-ca" pod "6f03dc4b-361e-419b-a1ae-46a2146e0e98" (UID: "6f03dc4b-361e-419b-a1ae-46a2146e0e98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:19.210918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.210882    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a719e5-b8cd-421a-9869-83b39f95391f-maas-api-service-ca" pod "84a719e5-b8cd-421a-9869-83b39f95391f" (UID: "84a719e5-b8cd-421a-9869-83b39f95391f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:19.211097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.211077    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a719e5-b8cd-421a-9869-83b39f95391f-maas-api-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.211097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.211097    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f03dc4b-361e-419b-a1ae-46a2146e0e98-maas-api-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.212750 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.212718    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f03dc4b-361e-419b-a1ae-46a2146e0e98-kube-api-access-w7q68" pod "6f03dc4b-361e-419b-a1ae-46a2146e0e98" (UID: "6f03dc4b-361e-419b-a1ae-46a2146e0e98"). InnerVolumeSpecName "kube-api-access-w7q68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:19.212835 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.212752    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a719e5-b8cd-421a-9869-83b39f95391f-kube-api-access-9cnhd" pod "84a719e5-b8cd-421a-9869-83b39f95391f" (UID: "84a719e5-b8cd-421a-9869-83b39f95391f"). InnerVolumeSpecName "kube-api-access-9cnhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:19.312545 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.312511    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cnhd\" (UniqueName: \"kubernetes.io/projected/84a719e5-b8cd-421a-9869-83b39f95391f-kube-api-access-9cnhd\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.312545 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.312539    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7q68\" (UniqueName: \"kubernetes.io/projected/6f03dc4b-361e-419b-a1ae-46a2146e0e98-kube-api-access-w7q68\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.444556 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.444520    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"]
Oct 30 19:15:19.449288 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.449260    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889795-ccm8w"]
Oct 30 19:15:19.963594 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.963562    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"
Oct 30 19:15:19.963804 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.963608    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz" event={"ID":"6f03dc4b-361e-419b-a1ae-46a2146e0e98","Type":"ContainerDied","Data":"a31f7fb392a68895703cff4df6147e56235d82459e13810a47f06fdef2c04427"}
Oct 30 19:15:19.963804 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.963693    4391 scope.go:122] "RemoveContainer" containerID="c014d188cc1b07467fa5b35da9d16e65ae71ef3e27a9264582577806f01e3b91"
Oct 30 19:15:19.965208 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.965182    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"
Oct 30 19:15:19.965320 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.965225    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd" event={"ID":"84a719e5-b8cd-421a-9869-83b39f95391f","Type":"ContainerDied","Data":"f0a2824e31885d804d783fc798e7338d3be40c15490b80cea6091f51c827cb66"}
Oct 30 19:15:19.975448 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:19.975431    4391 scope.go:122] "RemoveContainer" containerID="9bea4fea90f3045c5319109c70386c216a2777ee21a8be4fb7d3588b754391fb"
Oct 30 19:15:20.121039 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:20.121007    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"]
Oct 30 19:15:20.127300 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:20.127266    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889795-ftjnd"]
Oct 30 19:15:20.230392 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:20.230304    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"]
Oct 30 19:15:20.233668 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:20.233641    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-75e91b-29889795-szxbz"]
Oct 30 19:15:21.017629 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:21.017572    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" path="/var/lib/kubelet/pods/1bc3da2e-6fb6-4e35-9777-a57f5ccb0019/volumes"
Oct 30 19:15:21.018869 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:21.018842    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" path="/var/lib/kubelet/pods/6f03dc4b-361e-419b-a1ae-46a2146e0e98/volumes"
Oct 30 19:15:21.019544 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:21.019523    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a719e5-b8cd-421a-9869-83b39f95391f" path="/var/lib/kubelet/pods/84a719e5-b8cd-421a-9869-83b39f95391f/volumes"
Oct 30 19:15:21.020567 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:21.020546    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" path="/var/lib/kubelet/pods/ca127f30-3020-42f7-af2d-d23ca6c79fcb/volumes"
Oct 30 19:15:26.051230 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.051175    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"]
Oct 30 19:15:26.051785 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.051466    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh" podUID="2b3537b0-91de-458c-b368-50e83eb38ee1" containerName="maas-api" containerID="cri-o://53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2" gracePeriod=30
Oct 30 19:15:26.329010 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.328971    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:15:26.469234 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.469195    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-maas-api-tls\") pod \"2b3537b0-91de-458c-b368-50e83eb38ee1\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") "
Oct 30 19:15:26.469433 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.469251    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3537b0-91de-458c-b368-50e83eb38ee1-kube-api-access-bjmkt\" (UniqueName: \"kubernetes.io/projected/2b3537b0-91de-458c-b368-50e83eb38ee1-kube-api-access-bjmkt\") pod \"2b3537b0-91de-458c-b368-50e83eb38ee1\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") "
Oct 30 19:15:26.469593 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.469547    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b3537b0-91de-458c-b368-50e83eb38ee1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3537b0-91de-458c-b368-50e83eb38ee1-openshift-service-ca\") pod \"2b3537b0-91de-458c-b368-50e83eb38ee1\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") "
Oct 30 19:15:26.469735 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.469630    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-metrics-certs\") pod \"2b3537b0-91de-458c-b368-50e83eb38ee1\" (UID: \"2b3537b0-91de-458c-b368-50e83eb38ee1\") "
Oct 30 19:15:26.470207 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.469933    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3537b0-91de-458c-b368-50e83eb38ee1-openshift-service-ca" pod "2b3537b0-91de-458c-b368-50e83eb38ee1" (UID: "2b3537b0-91de-458c-b368-50e83eb38ee1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:26.470301 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.470241    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3537b0-91de-458c-b368-50e83eb38ee1-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:26.472718 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.472696    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3537b0-91de-458c-b368-50e83eb38ee1-kube-api-access-bjmkt" pod "2b3537b0-91de-458c-b368-50e83eb38ee1" (UID: "2b3537b0-91de-458c-b368-50e83eb38ee1"). InnerVolumeSpecName "kube-api-access-bjmkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:26.473357 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.473331    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-metrics-certs" pod "2b3537b0-91de-458c-b368-50e83eb38ee1" (UID: "2b3537b0-91de-458c-b368-50e83eb38ee1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:26.475025 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.474999    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-maas-api-tls" pod "2b3537b0-91de-458c-b368-50e83eb38ee1" (UID: "2b3537b0-91de-458c-b368-50e83eb38ee1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:26.571145 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.571056    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:26.571145 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.571090    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3537b0-91de-458c-b368-50e83eb38ee1-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:26.571145 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:26.571101    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmkt\" (UniqueName: \"kubernetes.io/projected/2b3537b0-91de-458c-b368-50e83eb38ee1-kube-api-access-bjmkt\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.021139 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:27.021108    4391 generic.go:356] "Generic (PLEG): container finished" podID="2b3537b0-91de-458c-b368-50e83eb38ee1" containerID="53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2" exitCode=0
Oct 30 19:15:27.021318 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:27.021147    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"
Oct 30 19:15:27.021318 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:27.021189    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh" event={"ID":"2b3537b0-91de-458c-b368-50e83eb38ee1","Type":"ContainerDied","Data":"53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2"}
Oct 30 19:15:27.021318 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:27.021238    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh" event={"ID":"2b3537b0-91de-458c-b368-50e83eb38ee1","Type":"ContainerDied","Data":"d7864bebd23244c99d46125dd9b4e264d2aaa5964547b606a36206f82ee19d63"}
Oct 30 19:15:27.021318 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:27.021260    4391 scope.go:122] "RemoveContainer" containerID="53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2"
Oct 30 19:15:27.031420 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:27.031399    4391 scope.go:122] "RemoveContainer" containerID="53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2"
Oct 30 19:15:27.031736 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:27.031705    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2\": container with ID starting with 53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2 not found: ID does not exist" containerID="53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2"
Oct 30 19:15:27.031809 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:27.031746    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2"} err="failed to get container status \"53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2\": rpc error: code = NotFound desc = could not find container \"53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2\": container with ID starting with 53b16c3af6a84eb526bb40e39201f81645d6ce7a09300d50488dbf6d015a30e2 not found: ID does not exist"
Oct 30 19:15:27.197119 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:27.197076    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"]
Oct 30 19:15:27.240334 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:27.240300    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-568107-bc7bd6f8d-lxvwh"]
Oct 30 19:15:29.008890 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:29.008856    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b3537b0-91de-458c-b368-50e83eb38ee1" path="/var/lib/kubelet/pods/2b3537b0-91de-458c-b368-50e83eb38ee1/volumes"
Oct 30 19:15:32.541846 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.541801    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" containerID="cri-o://01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074" gracePeriod=45
Oct 30 19:15:32.569873 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.569842    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 30 19:15:32.840296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.840267    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 19:15:32.902006 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.901965    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-model-cache\") pod \"7527b17f-8e81-44eb-8d5e-933530939eca\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") "
Oct 30 19:15:32.902006 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.902012    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-home\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-home\") pod \"7527b17f-8e81-44eb-8d5e-933530939eca\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") "
Oct 30 19:15:32.902252 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.902116    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-tmp-dir\") pod \"7527b17f-8e81-44eb-8d5e-933530939eca\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") "
Oct 30 19:15:32.902317 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.902284    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-model-cache" pod "7527b17f-8e81-44eb-8d5e-933530939eca" (UID: "7527b17f-8e81-44eb-8d5e-933530939eca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:32.902317 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.902291    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-dshm\") pod \"7527b17f-8e81-44eb-8d5e-933530939eca\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") "
Oct 30 19:15:32.902425 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.902298    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-home" pod "7527b17f-8e81-44eb-8d5e-933530939eca" (UID: "7527b17f-8e81-44eb-8d5e-933530939eca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:32.902425 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.902407    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-tmp-dir" pod "7527b17f-8e81-44eb-8d5e-933530939eca" (UID: "7527b17f-8e81-44eb-8d5e-933530939eca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:32.902537 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.902398    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7527b17f-8e81-44eb-8d5e-933530939eca-kube-api-access-t5j9k\" (UniqueName: \"kubernetes.io/projected/7527b17f-8e81-44eb-8d5e-933530939eca-kube-api-access-t5j9k\") pod \"7527b17f-8e81-44eb-8d5e-933530939eca\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") "
Oct 30 19:15:32.902537 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.902512    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7527b17f-8e81-44eb-8d5e-933530939eca-tls-certs\" (UniqueName: \"kubernetes.io/secret/7527b17f-8e81-44eb-8d5e-933530939eca-tls-certs\") pod \"7527b17f-8e81-44eb-8d5e-933530939eca\" (UID: \"7527b17f-8e81-44eb-8d5e-933530939eca\") "
Oct 30 19:15:32.903077 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.903034    4391 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-model-cache\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:32.903077 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.903056    4391 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-home\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:32.903077 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.903073    4391 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-tmp-dir\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:32.904532 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.904507    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-dshm" pod "7527b17f-8e81-44eb-8d5e-933530939eca" (UID: "7527b17f-8e81-44eb-8d5e-933530939eca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:32.904853 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.904830    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7527b17f-8e81-44eb-8d5e-933530939eca-tls-certs" pod "7527b17f-8e81-44eb-8d5e-933530939eca" (UID: "7527b17f-8e81-44eb-8d5e-933530939eca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:32.904966 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:32.904851    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7527b17f-8e81-44eb-8d5e-933530939eca-kube-api-access-t5j9k" pod "7527b17f-8e81-44eb-8d5e-933530939eca" (UID: "7527b17f-8e81-44eb-8d5e-933530939eca"). InnerVolumeSpecName "kube-api-access-t5j9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:33.005097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.005066    4391 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7527b17f-8e81-44eb-8d5e-933530939eca-dshm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.005097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.005097    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5j9k\" (UniqueName: \"kubernetes.io/projected/7527b17f-8e81-44eb-8d5e-933530939eca-kube-api-access-t5j9k\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.005292 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.005112    4391 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7527b17f-8e81-44eb-8d5e-933530939eca-tls-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.066246 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.066157    4391 generic.go:356] "Generic (PLEG): container finished" podID="7527b17f-8e81-44eb-8d5e-933530939eca" containerID="01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074" exitCode=0
Oct 30 19:15:33.066421 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.066397    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" event={"ID":"7527b17f-8e81-44eb-8d5e-933530939eca","Type":"ContainerDied","Data":"01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074"}
Oct 30 19:15:33.066551 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.066437    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6" event={"ID":"7527b17f-8e81-44eb-8d5e-933530939eca","Type":"ContainerDied","Data":"076564d2ac33406b350d3043e5458123dbbcf5afe7cca6c774ae6892c2dc8da7"}
Oct 30 19:15:33.066551 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.066474    4391 scope.go:122] "RemoveContainer" containerID="01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074"
Oct 30 19:15:33.066731 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.066627    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"
Oct 30 19:15:33.089071 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.089041    4391 scope.go:122] "RemoveContainer" containerID="01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074"
Oct 30 19:15:33.090295 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:33.090251    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074\": container with ID starting with 01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074 not found: ID does not exist" containerID="01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074"
Oct 30 19:15:33.090502 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.090472    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074"} err="failed to get container status \"01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074\": rpc error: code = NotFound desc = could not find container \"01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074\": container with ID starting with 01ce90ab042d13dbd989fe3e45087c1b3c2c025f536c45be1052b68e7a32c074 not found: ID does not exist"
Oct 30 19:15:33.229144 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.229076    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"]
Oct 30 19:15:33.233141 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.233114    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-w1-5f80ac-kserve-67468c9747-vl6n6"]
Oct 30 19:15:33.404271 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:33.404185    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:15:35.096742 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:35.096704    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" path="/var/lib/kubelet/pods/7527b17f-8e81-44eb-8d5e-933530939eca/volumes"
Oct 30 19:15:39.766866 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:39.766828    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"]
Oct 30 19:15:39.767505 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:39.767092    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98" podUID="f045ef47-c2dc-4e15-b71e-ae1d5de0342c" containerName="maas-api" containerID="cri-o://bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0" gracePeriod=30
Oct 30 19:15:40.113380 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.113344    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:15:40.203604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.199002    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-maas-api-tls\") pod \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") "
Oct 30 19:15:40.203604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.199159    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-metrics-certs\") pod \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") "
Oct 30 19:15:40.203604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.199296    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-kube-api-access-9dgvv\" (UniqueName: \"kubernetes.io/projected/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-kube-api-access-9dgvv\") pod \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") "
Oct 30 19:15:40.203604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.199369    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-openshift-service-ca\") pod \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\" (UID: \"f045ef47-c2dc-4e15-b71e-ae1d5de0342c\") "
Oct 30 19:15:40.204147 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.204113    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-openshift-service-ca" pod "f045ef47-c2dc-4e15-b71e-ae1d5de0342c" (UID: "f045ef47-c2dc-4e15-b71e-ae1d5de0342c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:40.214419 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.214387    4391 generic.go:356] "Generic (PLEG): container finished" podID="f045ef47-c2dc-4e15-b71e-ae1d5de0342c" containerID="bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0" exitCode=0
Oct 30 19:15:40.214690 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.214668    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98" event={"ID":"f045ef47-c2dc-4e15-b71e-ae1d5de0342c","Type":"ContainerDied","Data":"bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0"}
Oct 30 19:15:40.214825 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.214810    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98" event={"ID":"f045ef47-c2dc-4e15-b71e-ae1d5de0342c","Type":"ContainerDied","Data":"78499693160fcebcdefbfbbafec7e62f3881b6bb5d97fa33db1163843fca1d2a"}
Oct 30 19:15:40.218433 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.214929    4391 scope.go:122] "RemoveContainer" containerID="bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0"
Oct 30 19:15:40.219095 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.219075    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"
Oct 30 19:15:40.221825 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.221793    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-maas-api-tls" pod "f045ef47-c2dc-4e15-b71e-ae1d5de0342c" (UID: "f045ef47-c2dc-4e15-b71e-ae1d5de0342c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:40.227850 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.227801    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-metrics-certs" pod "f045ef47-c2dc-4e15-b71e-ae1d5de0342c" (UID: "f045ef47-c2dc-4e15-b71e-ae1d5de0342c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:40.228154 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.228115    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-kube-api-access-9dgvv" pod "f045ef47-c2dc-4e15-b71e-ae1d5de0342c" (UID: "f045ef47-c2dc-4e15-b71e-ae1d5de0342c"). InnerVolumeSpecName "kube-api-access-9dgvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:40.266157 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.266113    4391 scope.go:122] "RemoveContainer" containerID="bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0"
Oct 30 19:15:40.266799 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:40.266772    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0\": container with ID starting with bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0 not found: ID does not exist" containerID="bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0"
Oct 30 19:15:40.266933 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.266807    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0"} err="failed to get container status \"bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0\": rpc error: code = NotFound desc = could not find container \"bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0\": container with ID starting with bfb85c81e6b81e029ddbb3bf26d9639fe41660459607f69efe5fb37668ff6bf0 not found: ID does not exist"
Oct 30 19:15:40.300965 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.300926    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:40.300965 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.300956    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:40.300965 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.300967    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dgvv\" (UniqueName: \"kubernetes.io/projected/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-kube-api-access-9dgvv\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:40.301275 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.300978    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f045ef47-c2dc-4e15-b71e-ae1d5de0342c-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:40.639778 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.639740    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"]
Oct 30 19:15:40.659951 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:40.659920    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-314bd6fa-65c588785b-8mf98"]
Oct 30 19:15:41.006815 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:41.006784    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f045ef47-c2dc-4e15-b71e-ae1d5de0342c" path="/var/lib/kubelet/pods/f045ef47-c2dc-4e15-b71e-ae1d5de0342c/volumes"
Oct 30 19:15:41.253762 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:41.253725    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:15:43.714514 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:43.714458    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"]
Oct 30 19:15:47.447514 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:47.447479    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"]
Oct 30 19:15:50.768090 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:50.768051    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"]
Oct 30 19:15:55.769286 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:55.769243    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5" podUID="71648928-19f6-4214-b199-9b336e6bde04" containerName="payload-pre-processing" containerID="cri-o://828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21" gracePeriod=25
Oct 30 19:15:56.304492 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:56.304453    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5" podUID="71648928-19f6-4214-b199-9b336e6bde04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:58.751815 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:58.751750    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main" containerID="cri-o://4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f" gracePeriod=45
Oct 30 19:15:59.028929 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.028901    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 19:15:59.215503 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.215467    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-model-cache\") pod \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") "
Oct 30 19:15:59.215704 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.215549    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-home\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-home\") pod \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") "
Oct 30 19:15:59.215704 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.215651    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-kube-api-access-bmxd9\" (UniqueName: \"kubernetes.io/projected/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-kube-api-access-bmxd9\") pod \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") "
Oct 30 19:15:59.215828 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.215730    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tls-certs\" (UniqueName: \"kubernetes.io/secret/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tls-certs\") pod \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") "
Oct 30 19:15:59.215828 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.215760    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-dshm\") pod \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") "
Oct 30 19:15:59.215828 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.215758    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-model-cache" pod "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" (UID: "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.215828 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.215798    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-home" pod "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" (UID: "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.216030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.215860    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tmp-dir\") pod \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\" (UID: \"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10\") "
Oct 30 19:15:59.216099 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.216076    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tmp-dir" pod "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" (UID: "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.216344 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.216325    4391 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tmp-dir\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.216435 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.216346    4391 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-model-cache\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.216435 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.216361    4391 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-home\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.217982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.217957    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-dshm" pod "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" (UID: "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.218110 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.217988    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tls-certs" pod "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" (UID: "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:59.218110 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.218006    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-kube-api-access-bmxd9" pod "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" (UID: "065fbb95-3d5b-47fb-8daa-bc9cc33a5b10"). InnerVolumeSpecName "kube-api-access-bmxd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:59.317486 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.317399    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmxd9\" (UniqueName: \"kubernetes.io/projected/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-kube-api-access-bmxd9\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.317486 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.317432    4391 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-tls-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.317486 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.317441    4391 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10-dshm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.362955 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.362917    4391 generic.go:356] "Generic (PLEG): container finished" podID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerID="4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f" exitCode=0
Oct 30 19:15:59.363126 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.362964    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"
Oct 30 19:15:59.363126 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.363001    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" event={"ID":"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10","Type":"ContainerDied","Data":"4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f"}
Oct 30 19:15:59.363126 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.363045    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6" event={"ID":"065fbb95-3d5b-47fb-8daa-bc9cc33a5b10","Type":"ContainerDied","Data":"50ec1c88352be87f4171ec44216d2b131e38a5c6ddeb23e3e762e64a86c82faa"}
Oct 30 19:15:59.363126 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.363064    4391 scope.go:122] "RemoveContainer" containerID="4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f"
Oct 30 19:15:59.373480 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.373462    4391 scope.go:122] "RemoveContainer" containerID="4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f"
Oct 30 19:15:59.373773 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:15:59.373749    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f\": container with ID starting with 4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f not found: ID does not exist" containerID="4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f"
Oct 30 19:15:59.373829 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.373782    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f"} err="failed to get container status \"4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f\": rpc error: code = NotFound desc = could not find container \"4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f\": container with ID starting with 4d126a6745e7dbf549d491072ba1371406152d64fa5d403ea35f06d5c3c96f0f not found: ID does not exist"
Oct 30 19:15:59.520373 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.520336    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"]
Oct 30 19:15:59.566952 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:15:59.566920    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/premium-w3-a1617e-kserve-6695fcbc9d-xcvn6"]
Oct 30 19:16:01.006986 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:01.006956    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" path="/var/lib/kubelet/pods/065fbb95-3d5b-47fb-8daa-bc9cc33a5b10/volumes"
Oct 30 19:16:02.469046 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.469002    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x" podUID="a1dc6b43-837b-4600-b34e-76cf341240bd" containerName="main" containerID="cri-o://de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184" gracePeriod=45
Oct 30 19:16:02.731040 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.730978    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:16:02.761405 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.761368    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1dc6b43-837b-4600-b34e-76cf341240bd-kube-api-access-wslr6\" (UniqueName: \"kubernetes.io/projected/a1dc6b43-837b-4600-b34e-76cf341240bd-kube-api-access-wslr6\") pod \"a1dc6b43-837b-4600-b34e-76cf341240bd\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") "
Oct 30 19:16:02.761405 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.761406    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-tmp-dir\") pod \"a1dc6b43-837b-4600-b34e-76cf341240bd\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") "
Oct 30 19:16:02.761689 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.761501    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-home\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-home\") pod \"a1dc6b43-837b-4600-b34e-76cf341240bd\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") "
Oct 30 19:16:02.761689 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.761532    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1dc6b43-837b-4600-b34e-76cf341240bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1dc6b43-837b-4600-b34e-76cf341240bd-tls-certs\") pod \"a1dc6b43-837b-4600-b34e-76cf341240bd\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") "
Oct 30 19:16:02.761689 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.761561    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-model-cache\") pod \"a1dc6b43-837b-4600-b34e-76cf341240bd\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") "
Oct 30 19:16:02.761857 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.761746    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-dshm\") pod \"a1dc6b43-837b-4600-b34e-76cf341240bd\" (UID: \"a1dc6b43-837b-4600-b34e-76cf341240bd\") "
Oct 30 19:16:02.761930 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.761843    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-tmp-dir" pod "a1dc6b43-837b-4600-b34e-76cf341240bd" (UID: "a1dc6b43-837b-4600-b34e-76cf341240bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:02.761930 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.761827    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-home" pod "a1dc6b43-837b-4600-b34e-76cf341240bd" (UID: "a1dc6b43-837b-4600-b34e-76cf341240bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:02.761930 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.761896    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-model-cache" pod "a1dc6b43-837b-4600-b34e-76cf341240bd" (UID: "a1dc6b43-837b-4600-b34e-76cf341240bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:02.762206 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.762189    4391 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-model-cache\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:02.762269 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.762207    4391 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-tmp-dir\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:02.762269 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.762221    4391 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-home\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:02.763863 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.763834    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-dshm" pod "a1dc6b43-837b-4600-b34e-76cf341240bd" (UID: "a1dc6b43-837b-4600-b34e-76cf341240bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:02.763967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.763866    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1dc6b43-837b-4600-b34e-76cf341240bd-tls-certs" pod "a1dc6b43-837b-4600-b34e-76cf341240bd" (UID: "a1dc6b43-837b-4600-b34e-76cf341240bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:02.763967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.763905    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1dc6b43-837b-4600-b34e-76cf341240bd-kube-api-access-wslr6" pod "a1dc6b43-837b-4600-b34e-76cf341240bd" (UID: "a1dc6b43-837b-4600-b34e-76cf341240bd"). InnerVolumeSpecName "kube-api-access-wslr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:02.863839 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.863794    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wslr6\" (UniqueName: \"kubernetes.io/projected/a1dc6b43-837b-4600-b34e-76cf341240bd-kube-api-access-wslr6\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:02.863839 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.863828    4391 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1dc6b43-837b-4600-b34e-76cf341240bd-tls-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:02.863839 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:02.863837    4391 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1dc6b43-837b-4600-b34e-76cf341240bd-dshm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:03.390151 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:03.390070    4391 generic.go:356] "Generic (PLEG): container finished" podID="a1dc6b43-837b-4600-b34e-76cf341240bd" containerID="de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184" exitCode=0
Oct 30 19:16:03.390151 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:03.390135    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"
Oct 30 19:16:03.390350 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:03.390148    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x" event={"ID":"a1dc6b43-837b-4600-b34e-76cf341240bd","Type":"ContainerDied","Data":"de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184"}
Oct 30 19:16:03.390350 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:03.390201    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x" event={"ID":"a1dc6b43-837b-4600-b34e-76cf341240bd","Type":"ContainerDied","Data":"2ffce981543990578d470308999e812e28bd78685c5387ae2e9a77b37fbbbe40"}
Oct 30 19:16:03.390350 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:03.390224    4391 scope.go:122] "RemoveContainer" containerID="de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184"
Oct 30 19:16:03.399735 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:03.399708    4391 scope.go:122] "RemoveContainer" containerID="de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184"
Oct 30 19:16:03.400039 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:03.400018    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184\": container with ID starting with de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184 not found: ID does not exist" containerID="de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184"
Oct 30 19:16:03.400127 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:03.400049    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184"} err="failed to get container status \"de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184\": rpc error: code = NotFound desc = could not find container \"de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184\": container with ID starting with de9279a97714c769ed8b0c9590d51cd69dad1290bb8d07324f7811167b7cf184 not found: ID does not exist"
Oct 30 19:16:03.551632 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:03.551536    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"]
Oct 30 19:16:03.598248 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:03.598214    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-314bd6fa/e2e-lifecycle-model-314bd6fa-kserve-7b8dc99b97-jqj4x"]
Oct 30 19:16:05.006792 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:05.006756    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1dc6b43-837b-4600-b34e-76cf341240bd" path="/var/lib/kubelet/pods/a1dc6b43-837b-4600-b34e-76cf341240bd/volumes"
Oct 30 19:16:06.304568 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:06.304528    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5" podUID="71648928-19f6-4214-b199-9b336e6bde04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:16:11.252693 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:11.252654    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:16:12.590100 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:12.590037    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"]
Oct 30 19:16:12.590640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:12.590342    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2" podUID="2057bc1a-888a-4706-83fb-1006e3954ada" containerName="maas-api" containerID="cri-o://f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f" gracePeriod=30
Oct 30 19:16:12.918666 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:12.918633    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:16:12.997539 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:12.997508    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-metrics-certs\") pod \"2057bc1a-888a-4706-83fb-1006e3954ada\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") "
Oct 30 19:16:12.997724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:12.997567    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2057bc1a-888a-4706-83fb-1006e3954ada-kube-api-access-g5rjm\" (UniqueName: \"kubernetes.io/projected/2057bc1a-888a-4706-83fb-1006e3954ada-kube-api-access-g5rjm\") pod \"2057bc1a-888a-4706-83fb-1006e3954ada\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") "
Oct 30 19:16:12.997724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:12.997637    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2057bc1a-888a-4706-83fb-1006e3954ada-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2057bc1a-888a-4706-83fb-1006e3954ada-openshift-service-ca\") pod \"2057bc1a-888a-4706-83fb-1006e3954ada\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") "
Oct 30 19:16:12.997724 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:12.997706    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-maas-api-tls\") pod \"2057bc1a-888a-4706-83fb-1006e3954ada\" (UID: \"2057bc1a-888a-4706-83fb-1006e3954ada\") "
Oct 30 19:16:12.998125 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:12.998084    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2057bc1a-888a-4706-83fb-1006e3954ada-openshift-service-ca" pod "2057bc1a-888a-4706-83fb-1006e3954ada" (UID: "2057bc1a-888a-4706-83fb-1006e3954ada"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:12.998279 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:12.998246    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2057bc1a-888a-4706-83fb-1006e3954ada-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:13.000126 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.000088    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-metrics-certs" pod "2057bc1a-888a-4706-83fb-1006e3954ada" (UID: "2057bc1a-888a-4706-83fb-1006e3954ada"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:13.000256 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.000219    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-maas-api-tls" pod "2057bc1a-888a-4706-83fb-1006e3954ada" (UID: "2057bc1a-888a-4706-83fb-1006e3954ada"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:13.000350 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.000324    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2057bc1a-888a-4706-83fb-1006e3954ada-kube-api-access-g5rjm" pod "2057bc1a-888a-4706-83fb-1006e3954ada" (UID: "2057bc1a-888a-4706-83fb-1006e3954ada"). InnerVolumeSpecName "kube-api-access-g5rjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:13.099050 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.099007    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:13.099050 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.099039    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rjm\" (UniqueName: \"kubernetes.io/projected/2057bc1a-888a-4706-83fb-1006e3954ada-kube-api-access-g5rjm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:13.099050 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.099055    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2057bc1a-888a-4706-83fb-1006e3954ada-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:13.485892 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.485855    4391 generic.go:356] "Generic (PLEG): container finished" podID="2057bc1a-888a-4706-83fb-1006e3954ada" containerID="f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f" exitCode=0
Oct 30 19:16:13.486097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.485910    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"
Oct 30 19:16:13.486097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.485946    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2" event={"ID":"2057bc1a-888a-4706-83fb-1006e3954ada","Type":"ContainerDied","Data":"f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f"}
Oct 30 19:16:13.486097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.485997    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2" event={"ID":"2057bc1a-888a-4706-83fb-1006e3954ada","Type":"ContainerDied","Data":"891e0908c0440f3d12928712c17babf678f051f872b8652ec1ad6d956fc4848f"}
Oct 30 19:16:13.486097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.486019    4391 scope.go:122] "RemoveContainer" containerID="f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f"
Oct 30 19:16:13.498391 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.498366    4391 scope.go:122] "RemoveContainer" containerID="f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f"
Oct 30 19:16:13.498696 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:13.498676    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f\": container with ID starting with f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f not found: ID does not exist" containerID="f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f"
Oct 30 19:16:13.498750 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.498705    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f"} err="failed to get container status \"f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f\": rpc error: code = NotFound desc = could not find container \"f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f\": container with ID starting with f08f4346699437b66ff22d28fee7697cb12c6950a2eeaf44e44782a2f385696f not found: ID does not exist"
Oct 30 19:16:13.648905 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.648836    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"]
Oct 30 19:16:13.693122 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:13.693086    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8e31a7-b6bfcbf44-dk7f2"]
Oct 30 19:16:15.007032 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:15.007002    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2057bc1a-888a-4706-83fb-1006e3954ada" path="/var/lib/kubelet/pods/2057bc1a-888a-4706-83fb-1006e3954ada/volumes"
Oct 30 19:16:16.032308 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.032245    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 19:16:16.041371 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.041347    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71648928-19f6-4214-b199-9b336e6bde04-kube-api-access-7ltww\" (UniqueName: \"kubernetes.io/projected/71648928-19f6-4214-b199-9b336e6bde04-kube-api-access-7ltww\") pod \"71648928-19f6-4214-b199-9b336e6bde04\" (UID: \"71648928-19f6-4214-b199-9b336e6bde04\") "
Oct 30 19:16:16.041475 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.041449    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71648928-19f6-4214-b199-9b336e6bde04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71648928-19f6-4214-b199-9b336e6bde04-plugins-config-volume\") pod \"71648928-19f6-4214-b199-9b336e6bde04\" (UID: \"71648928-19f6-4214-b199-9b336e6bde04\") "
Oct 30 19:16:16.041823 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.041800    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71648928-19f6-4214-b199-9b336e6bde04-plugins-config-volume" pod "71648928-19f6-4214-b199-9b336e6bde04" (UID: "71648928-19f6-4214-b199-9b336e6bde04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:16.041966 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.041951    4391 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71648928-19f6-4214-b199-9b336e6bde04-plugins-config-volume\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:16.043370 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.043347    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71648928-19f6-4214-b199-9b336e6bde04-kube-api-access-7ltww" pod "71648928-19f6-4214-b199-9b336e6bde04" (UID: "71648928-19f6-4214-b199-9b336e6bde04"). InnerVolumeSpecName "kube-api-access-7ltww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:16.142942 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.142907    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ltww\" (UniqueName: \"kubernetes.io/projected/71648928-19f6-4214-b199-9b336e6bde04-kube-api-access-7ltww\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:16.507626 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.507572    4391 generic.go:356] "Generic (PLEG): container finished" podID="71648928-19f6-4214-b199-9b336e6bde04" containerID="828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21" exitCode=0
Oct 30 19:16:16.507626 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.507605    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5" event={"ID":"71648928-19f6-4214-b199-9b336e6bde04","Type":"ContainerDied","Data":"828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21"}
Oct 30 19:16:16.507867 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.507648    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"
Oct 30 19:16:16.507867 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.507667    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5" event={"ID":"71648928-19f6-4214-b199-9b336e6bde04","Type":"ContainerDied","Data":"fb425742fdf5ad76cf5342cf1b08c1e3fcf92a239a24724f4541fcc74828803a"}
Oct 30 19:16:16.507867 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.507692    4391 scope.go:122] "RemoveContainer" containerID="828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21"
Oct 30 19:16:16.518622 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.518569    4391 scope.go:122] "RemoveContainer" containerID="828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21"
Oct 30 19:16:16.518869 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:16.518850    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21\": container with ID starting with 828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21 not found: ID does not exist" containerID="828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21"
Oct 30 19:16:16.518941 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.518878    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21"} err="failed to get container status \"828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21\": rpc error: code = NotFound desc = could not find container \"828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21\": container with ID starting with 828d1b3c1d39f5cb739bcc532f6239256985826713230d4f699316a2e3910e21 not found: ID does not exist"
Oct 30 19:16:16.668737 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.668701    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"]
Oct 30 19:16:16.711410 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:16.711377    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a1617e-7fd8d65f5f-rfsn5"]
Oct 30 19:16:17.005214 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:17.005179    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71648928-19f6-4214-b199-9b336e6bde04" path="/var/lib/kubelet/pods/71648928-19f6-4214-b199-9b336e6bde04/volumes"
Oct 30 19:16:19.723953 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.723916    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"]
Oct 30 19:16:19.725094 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725065    4391 cpu_manager.go:414] "containerMap: removing container" podUID="f045ef47-c2dc-4e15-b71e-ae1d5de0342c" containerName="maas-api"
Oct 30 19:16:19.725094 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725089    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f045ef47-c2dc-4e15-b71e-ae1d5de0342c" containerName="maas-api"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725102    4391 cpu_manager.go:414] "containerMap: removing container" podUID="84a719e5-b8cd-421a-9869-83b39f95391f" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725108    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a719e5-b8cd-421a-9869-83b39f95391f" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725118    4391 cpu_manager.go:414] "containerMap: removing container" podUID="71279088-1baf-4ac7-a020-a22ce6f5f76b" containerName="maas-api"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725122    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71279088-1baf-4ac7-a020-a22ce6f5f76b" containerName="maas-api"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725131    4391 cpu_manager.go:414] "containerMap: removing container" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725135    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725141    4391 cpu_manager.go:414] "containerMap: removing container" podUID="84a719e5-b8cd-421a-9869-83b39f95391f" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725145    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a719e5-b8cd-421a-9869-83b39f95391f" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725155    4391 cpu_manager.go:414] "containerMap: removing container" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725159    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725167    4391 cpu_manager.go:414] "containerMap: removing container" podUID="2057bc1a-888a-4706-83fb-1006e3954ada" containerName="maas-api"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725171    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2057bc1a-888a-4706-83fb-1006e3954ada" containerName="maas-api"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725178    4391 cpu_manager.go:414] "containerMap: removing container" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725182    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725190    4391 cpu_manager.go:414] "containerMap: removing container" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725194    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725200    4391 cpu_manager.go:414] "containerMap: removing container" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725204    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725212    4391 cpu_manager.go:414] "containerMap: removing container" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725217    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725226    4391 cpu_manager.go:414] "containerMap: removing container" podUID="71648928-19f6-4214-b199-9b336e6bde04" containerName="payload-pre-processing"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725231    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71648928-19f6-4214-b199-9b336e6bde04" containerName="payload-pre-processing"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725237    4391 cpu_manager.go:414] "containerMap: removing container" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725241    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725258    4391 cpu_manager.go:414] "containerMap: removing container" podUID="2b3537b0-91de-458c-b368-50e83eb38ee1" containerName="maas-api"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725262    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3537b0-91de-458c-b368-50e83eb38ee1" containerName="maas-api"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725273    4391 cpu_manager.go:414] "containerMap: removing container" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725277    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725282    4391 cpu_manager.go:414] "containerMap: removing container" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725287    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerName="cleanup"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725295    4391 cpu_manager.go:414] "containerMap: removing container" podUID="a1dc6b43-837b-4600-b34e-76cf341240bd" containerName="main"
Oct 30 19:16:19.725294 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725300    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1dc6b43-837b-4600-b34e-76cf341240bd" containerName="main"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725380    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerName="cleanup"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725388    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerName="cleanup"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725395    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1dc6b43-837b-4600-b34e-76cf341240bd" containerName="main"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725402    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerName="cleanup"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725408    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="f045ef47-c2dc-4e15-b71e-ae1d5de0342c" containerName="maas-api"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725415    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a719e5-b8cd-421a-9869-83b39f95391f" containerName="cleanup"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725420    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="71279088-1baf-4ac7-a020-a22ce6f5f76b" containerName="maas-api"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725425    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerName="cleanup"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725430    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a719e5-b8cd-421a-9869-83b39f95391f" containerName="cleanup"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725436    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerName="cleanup"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725441    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="2057bc1a-888a-4706-83fb-1006e3954ada" containerName="maas-api"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725445    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc3da2e-6fb6-4e35-9777-a57f5ccb0019" containerName="cleanup"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725450    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerName="cleanup"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725455    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="065fbb95-3d5b-47fb-8daa-bc9cc33a5b10" containerName="main"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725459    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="71648928-19f6-4214-b199-9b336e6bde04" containerName="payload-pre-processing"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725464    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="7527b17f-8e81-44eb-8d5e-933530939eca" containerName="main"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725471    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerName="cleanup"
Oct 30 19:16:19.726982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.725477    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3537b0-91de-458c-b368-50e83eb38ee1" containerName="maas-api"
Oct 30 19:16:19.731972 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.731937    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.781287 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.781251    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"]
Oct 30 19:16:19.797416 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.797376    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwnx\" (UniqueName: \"kubernetes.io/projected/61b5903c-5d65-4bd8-9018-5f0930e14260-kube-api-access-7xwnx\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.797651 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.797459    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5903c-5d65-4bd8-9018-5f0930e14260-openshift-service-ca\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.797651 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.797553    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-metrics-certs\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.797651 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.797646    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-maas-api-tls\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.824130 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.824100    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-29de7b-serving-cert\""
Oct 30 19:16:19.899178 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.899142    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwnx\" (UniqueName: \"kubernetes.io/projected/61b5903c-5d65-4bd8-9018-5f0930e14260-kube-api-access-7xwnx\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.899367 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.899205    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5903c-5d65-4bd8-9018-5f0930e14260-openshift-service-ca\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.899367 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.899285    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-metrics-certs\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.899367 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.899349    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-maas-api-tls\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.899533 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:19.899501    4391 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-serving-cert: secret "maas-api-e2e-inf-b-29de7b-serving-cert" not found
Oct 30 19:16:19.899662 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:19.899605    4391 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-maas-api-tls podName:61b5903c-5d65-4bd8-9018-5f0930e14260 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:20.399570445 +0000 UTC m=+2375.510303720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-maas-api-tls") pod "maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69" (UID: "61b5903c-5d65-4bd8-9018-5f0930e14260") : secret "maas-api-e2e-inf-b-29de7b-serving-cert" not found
Oct 30 19:16:19.900005 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.899980    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5903c-5d65-4bd8-9018-5f0930e14260-openshift-service-ca\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.901902 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.901875    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-metrics-certs\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:19.956805 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:19.956520    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwnx\" (UniqueName: \"kubernetes.io/projected/61b5903c-5d65-4bd8-9018-5f0930e14260-kube-api-access-7xwnx\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:20.408111 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:20.408057    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-maas-api-tls\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:20.411030 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:20.410998    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-maas-api-tls\") pod \"maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:20.647892 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:20.647856    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:21.028191 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:21.028166    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"]
Oct 30 19:16:21.030066 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:16:21.030043    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b5903c_5d65_4bd8_9018_5f0930e14260.slice/crio-77638d00ea28d00bc5371b1415a8d11a70d272ccfb19c728e7ca6bd1b1816174 WatchSource:0}: Error finding container 77638d00ea28d00bc5371b1415a8d11a70d272ccfb19c728e7ca6bd1b1816174: Status 404 returned error can't find the container with id 77638d00ea28d00bc5371b1415a8d11a70d272ccfb19c728e7ca6bd1b1816174
Oct 30 19:16:21.031717 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:21.031699    4391 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:16:21.549374 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:21.549333    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69" event={"ID":"61b5903c-5d65-4bd8-9018-5f0930e14260","Type":"ContainerStarted","Data":"77638d00ea28d00bc5371b1415a8d11a70d272ccfb19c728e7ca6bd1b1816174"}
Oct 30 19:16:22.560263 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:22.560218    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69" event={"ID":"61b5903c-5d65-4bd8-9018-5f0930e14260","Type":"ContainerStarted","Data":"7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be"}
Oct 30 19:16:22.560806 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:22.560316    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:22.650717 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:22.650683    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"]
Oct 30 19:16:22.650903 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:22.650883    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy" containerID="cri-o://e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6" gracePeriod=30
Oct 30 19:16:22.718702 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:22.718613    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69" podStartSLOduration=2.693559757 podStartE2EDuration="3.718595875s" totalImagesPullingTime="1.025036118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:22.715163788 +0000 UTC m=+2377.825897070" watchObservedRunningTime="2026-10-30 19:16:22.718595875 +0000 UTC m=+2377.829329153"
Oct 30 19:16:25.758025 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:25.757986    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"]
Oct 30 19:16:25.771629 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:25.758845    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d" podUID="c7ce2533-9241-4180-9675-fcbd391c3105" containerName="maas-api" containerID="cri-o://ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc" gracePeriod=30
Oct 30 19:16:26.112855 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.112827    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:16:26.347991 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.347916    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ce2533-9241-4180-9675-fcbd391c3105-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ce2533-9241-4180-9675-fcbd391c3105-openshift-service-ca\") pod \"c7ce2533-9241-4180-9675-fcbd391c3105\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") "
Oct 30 19:16:26.348217 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.348106    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-metrics-certs\") pod \"c7ce2533-9241-4180-9675-fcbd391c3105\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") "
Oct 30 19:16:26.348217 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.348170    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ce2533-9241-4180-9675-fcbd391c3105-kube-api-access-lpb7x\" (UniqueName: \"kubernetes.io/projected/c7ce2533-9241-4180-9675-fcbd391c3105-kube-api-access-lpb7x\") pod \"c7ce2533-9241-4180-9675-fcbd391c3105\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") "
Oct 30 19:16:26.348307 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.348287    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ce2533-9241-4180-9675-fcbd391c3105-openshift-service-ca" pod "c7ce2533-9241-4180-9675-fcbd391c3105" (UID: "c7ce2533-9241-4180-9675-fcbd391c3105"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:26.348357 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.348304    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-maas-api-tls\") pod \"c7ce2533-9241-4180-9675-fcbd391c3105\" (UID: \"c7ce2533-9241-4180-9675-fcbd391c3105\") "
Oct 30 19:16:26.348813 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.348790    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ce2533-9241-4180-9675-fcbd391c3105-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:26.350323 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.350291    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-metrics-certs" pod "c7ce2533-9241-4180-9675-fcbd391c3105" (UID: "c7ce2533-9241-4180-9675-fcbd391c3105"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:26.350426 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.350346    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-maas-api-tls" pod "c7ce2533-9241-4180-9675-fcbd391c3105" (UID: "c7ce2533-9241-4180-9675-fcbd391c3105"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:26.350426 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.350398    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ce2533-9241-4180-9675-fcbd391c3105-kube-api-access-lpb7x" pod "c7ce2533-9241-4180-9675-fcbd391c3105" (UID: "c7ce2533-9241-4180-9675-fcbd391c3105"). InnerVolumeSpecName "kube-api-access-lpb7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:26.449630 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.449566    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:26.449630 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.449618    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ce2533-9241-4180-9675-fcbd391c3105-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:26.449630 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.449633    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpb7x\" (UniqueName: \"kubernetes.io/projected/c7ce2533-9241-4180-9675-fcbd391c3105-kube-api-access-lpb7x\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:26.591641 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.591572    4391 generic.go:356] "Generic (PLEG): container finished" podID="c7ce2533-9241-4180-9675-fcbd391c3105" containerID="ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc" exitCode=0
Oct 30 19:16:26.591641 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.591640    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"
Oct 30 19:16:26.591877 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.591650    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d" event={"ID":"c7ce2533-9241-4180-9675-fcbd391c3105","Type":"ContainerDied","Data":"ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc"}
Oct 30 19:16:26.591877 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.591702    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d" event={"ID":"c7ce2533-9241-4180-9675-fcbd391c3105","Type":"ContainerDied","Data":"8bde309278cedaed7fb9e0e420a80a2e785ec2f03cb38b50d93f042566f1f365"}
Oct 30 19:16:26.591877 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.591731    4391 scope.go:122] "RemoveContainer" containerID="ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc"
Oct 30 19:16:26.606317 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.606292    4391 scope.go:122] "RemoveContainer" containerID="ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc"
Oct 30 19:16:26.606799 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:26.606771    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc\": container with ID starting with ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc not found: ID does not exist" containerID="ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc"
Oct 30 19:16:26.606913 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.606807    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc"} err="failed to get container status \"ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc\": rpc error: code = NotFound desc = could not find container \"ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc\": container with ID starting with ac92635a6d15c27c20df1e49d1dee53bfb0052537c427672c88a369d3feda8dc not found: ID does not exist"
Oct 30 19:16:26.750509 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.750477    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"]
Oct 30 19:16:26.797732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:26.797690    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3a0e7-b8c47f9fc-mgh2d"]
Oct 30 19:16:27.006676 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.006633    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ce2533-9241-4180-9675-fcbd391c3105" path="/var/lib/kubelet/pods/c7ce2533-9241-4180-9675-fcbd391c3105/volumes"
Oct 30 19:16:27.921861 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.921801    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:16:27.989960 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.989924    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-kube-api-access-p2wlj\" (UniqueName: \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-kube-api-access-p2wlj\") pod \"b5514f78-b153-49dc-9d09-9ba5e104ce78\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") "
Oct 30 19:16:27.990133 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.989987    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-token\" (UniqueName: \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-token\") pod \"b5514f78-b153-49dc-9d09-9ba5e104ce78\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") "
Oct 30 19:16:27.990133 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990025    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-envoy\") pod \"b5514f78-b153-49dc-9d09-9ba5e104ce78\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") "
Oct 30 19:16:27.990133 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990085    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-podinfo\") pod \"b5514f78-b153-49dc-9d09-9ba5e104ce78\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") "
Oct 30 19:16:27.990268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990133    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-credential-socket\") pod \"b5514f78-b153-49dc-9d09-9ba5e104ce78\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") "
Oct 30 19:16:27.990268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990170    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-data\") pod \"b5514f78-b153-49dc-9d09-9ba5e104ce78\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") "
Oct 30 19:16:27.990268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990218    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-certs\") pod \"b5514f78-b153-49dc-9d09-9ba5e104ce78\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") "
Oct 30 19:16:27.990268 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990252    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5514f78-b153-49dc-9d09-9ba5e104ce78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5514f78-b153-49dc-9d09-9ba5e104ce78-istiod-ca-cert\") pod \"b5514f78-b153-49dc-9d09-9ba5e104ce78\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") "
Oct 30 19:16:27.990479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990303    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-socket\") pod \"b5514f78-b153-49dc-9d09-9ba5e104ce78\" (UID: \"b5514f78-b153-49dc-9d09-9ba5e104ce78\") "
Oct 30 19:16:27.990538 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990489    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-credential-socket" pod "b5514f78-b153-49dc-9d09-9ba5e104ce78" (UID: "b5514f78-b153-49dc-9d09-9ba5e104ce78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:27.990538 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990502    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-data" pod "b5514f78-b153-49dc-9d09-9ba5e104ce78" (UID: "b5514f78-b153-49dc-9d09-9ba5e104ce78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:27.990825 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990619    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-certs" pod "b5514f78-b153-49dc-9d09-9ba5e104ce78" (UID: "b5514f78-b153-49dc-9d09-9ba5e104ce78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:27.990825 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990757    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-socket" pod "b5514f78-b153-49dc-9d09-9ba5e104ce78" (UID: "b5514f78-b153-49dc-9d09-9ba5e104ce78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:27.990997 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.990879    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5514f78-b153-49dc-9d09-9ba5e104ce78-istiod-ca-cert" pod "b5514f78-b153-49dc-9d09-9ba5e104ce78" (UID: "b5514f78-b153-49dc-9d09-9ba5e104ce78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:27.991152 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.991123    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:27.991152 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.991153    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:27.991340 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.991167    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:27.991340 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.991181    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5514f78-b153-49dc-9d09-9ba5e104ce78-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:27.991340 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.991195    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:27.992537 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.992513    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-kube-api-access-p2wlj" pod "b5514f78-b153-49dc-9d09-9ba5e104ce78" (UID: "b5514f78-b153-49dc-9d09-9ba5e104ce78"). InnerVolumeSpecName "kube-api-access-p2wlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:27.992653 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.992547    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-token" pod "b5514f78-b153-49dc-9d09-9ba5e104ce78" (UID: "b5514f78-b153-49dc-9d09-9ba5e104ce78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:27.992738 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.992712    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-envoy" pod "b5514f78-b153-49dc-9d09-9ba5e104ce78" (UID: "b5514f78-b153-49dc-9d09-9ba5e104ce78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:27.992829 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:27.992810    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-podinfo" pod "b5514f78-b153-49dc-9d09-9ba5e104ce78" (UID: "b5514f78-b153-49dc-9d09-9ba5e104ce78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:28.092481 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.092447    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:28.092481 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.092474    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:28.092481 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.092486    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2wlj\" (UniqueName: \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-kube-api-access-p2wlj\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:28.092754 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.092496    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5514f78-b153-49dc-9d09-9ba5e104ce78-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:28.573275 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.573244    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:16:28.609141 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.609110    4391 generic.go:356] "Generic (PLEG): container finished" podID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerID="e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6" exitCode=0
Oct 30 19:16:28.609328 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.609153    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"
Oct 30 19:16:28.609328 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.609188    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" event={"ID":"b5514f78-b153-49dc-9d09-9ba5e104ce78","Type":"ContainerDied","Data":"e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6"}
Oct 30 19:16:28.609328 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.609250    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv" event={"ID":"b5514f78-b153-49dc-9d09-9ba5e104ce78","Type":"ContainerDied","Data":"08b237cb9dd27cc4901eb7cb3e32bc180b44e3192e1d5f62b570f162f6488faa"}
Oct 30 19:16:28.609328 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.609274    4391 scope.go:122] "RemoveContainer" containerID="e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6"
Oct 30 19:16:28.620737 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.620713    4391 scope.go:122] "RemoveContainer" containerID="e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6"
Oct 30 19:16:28.620998 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:28.620974    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6\": container with ID starting with e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6 not found: ID does not exist" containerID="e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6"
Oct 30 19:16:28.621066 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.621011    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6"} err="failed to get container status \"e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6\": rpc error: code = NotFound desc = could not find container \"e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6\": container with ID starting with e2329bcac4cc65a3064b3f711d50ff57420486bdae27e53fe223e8f72a024ab6 not found: ID does not exist"
Oct 30 19:16:28.804738 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.804706    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"]
Oct 30 19:16:28.809508 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:28.809476    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-l7kmv"]
Oct 30 19:16:29.005935 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:29.005885    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" path="/var/lib/kubelet/pods/b5514f78-b153-49dc-9d09-9ba5e104ce78/volumes"
Oct 30 19:16:29.764783 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:29.764749    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"]
Oct 30 19:16:33.914725 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:33.914690    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"]
Oct 30 19:16:33.915178 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:33.914934    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerName="istio-proxy" containerID="cri-o://7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae" gracePeriod=30
Oct 30 19:16:38.274605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:38.269158    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:16:39.274264 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.274241    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 19:16:39.387502 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.387470    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-token\" (UniqueName: \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-token\") pod \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") "
Oct 30 19:16:39.387967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.387546    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istiod-ca-cert\") pod \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") "
Oct 30 19:16:39.387967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.387666    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-certs\") pod \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") "
Oct 30 19:16:39.387967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.387722    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-credential-socket\") pod \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") "
Oct 30 19:16:39.387967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.387779    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-kube-api-access-rl4ws\" (UniqueName: \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-kube-api-access-rl4ws\") pod \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") "
Oct 30 19:16:39.387967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.387892    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-envoy\") pod \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") "
Oct 30 19:16:39.387967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.387918    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-socket\") pod \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") "
Oct 30 19:16:39.387967 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.387938    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-certs" pod "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" (UID: "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:39.388351 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388020    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istiod-ca-cert" pod "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" (UID: "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:39.388351 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388034    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-data\") pod \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") "
Oct 30 19:16:39.388351 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388125    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-credential-socket" pod "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" (UID: "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:39.388351 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388140    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-podinfo\") pod \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\" (UID: \"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261\") "
Oct 30 19:16:39.388351 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388226    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-socket" pod "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" (UID: "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:39.388542 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388372    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-data" pod "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" (UID: "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:39.388682 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388662    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.388732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388688    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.388732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388702    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.388732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388715    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.388732 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.388729    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.390106 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.390079    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-envoy" pod "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" (UID: "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:39.390318 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.390287    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-podinfo" pod "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" (UID: "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:39.390425 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.390297    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-token" pod "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" (UID: "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:39.390655 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.390635    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-kube-api-access-rl4ws" pod "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" (UID: "e85c4e22-99d6-4616-b2b1-3ce5bd0a8261"). InnerVolumeSpecName "kube-api-access-rl4ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:39.490358 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.490323    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.490358 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.490356    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl4ws\" (UniqueName: \"kubernetes.io/projected/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-kube-api-access-rl4ws\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.490553 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.490368    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.490553 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.490382    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.677624 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.677575    4391 generic.go:356] "Generic (PLEG): container finished" podID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerID="7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae" exitCode=0
Oct 30 19:16:39.677833 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.677643    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" event={"ID":"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261","Type":"ContainerDied","Data":"7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae"}
Oct 30 19:16:39.677833 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.677650    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"
Oct 30 19:16:39.677833 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.677696    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt" event={"ID":"e85c4e22-99d6-4616-b2b1-3ce5bd0a8261","Type":"ContainerDied","Data":"1e7693a3f0f0fed64c3be3aeb8e0d9e5451a117b00fbf5ef94bad9064986b293"}
Oct 30 19:16:39.677833 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.677725    4391 scope.go:122] "RemoveContainer" containerID="7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae"
Oct 30 19:16:39.688644 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.688620    4391 scope.go:122] "RemoveContainer" containerID="7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae"
Oct 30 19:16:39.688923 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:39.688904    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae\": container with ID starting with 7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae not found: ID does not exist" containerID="7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae"
Oct 30 19:16:39.688991 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.688932    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae"} err="failed to get container status \"7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae\": rpc error: code = NotFound desc = could not find container \"7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae\": container with ID starting with 7e4ad23306c166b3167c53a80f4ea220984fe27aee015b75d589fd00058311ae not found: ID does not exist"
Oct 30 19:16:39.843740 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.843702    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"]
Oct 30 19:16:39.850192 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:39.850167    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-8pcmt"]
Oct 30 19:16:41.006518 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:41.006480    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" path="/var/lib/kubelet/pods/e85c4e22-99d6-4616-b2b1-3ce5bd0a8261/volumes"
Oct 30 19:16:41.252660 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:41.252624    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:16:42.671457 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:42.671420    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"]
Oct 30 19:16:44.787018 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:44.786982    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main" containerID="cri-o://abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf" gracePeriod=45
Oct 30 19:16:45.039208 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.039149    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 19:16:45.056139 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.056109    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-model-cache\") pod \"ff3d235d-adec-4667-9553-3fe55500a0a8\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") "
Oct 30 19:16:45.056287 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.056159    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3d235d-adec-4667-9553-3fe55500a0a8-kube-api-access-rc7db\" (UniqueName: \"kubernetes.io/projected/ff3d235d-adec-4667-9553-3fe55500a0a8-kube-api-access-rc7db\") pod \"ff3d235d-adec-4667-9553-3fe55500a0a8\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") "
Oct 30 19:16:45.056287 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.056190    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff3d235d-adec-4667-9553-3fe55500a0a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3d235d-adec-4667-9553-3fe55500a0a8-tls-certs\") pod \"ff3d235d-adec-4667-9553-3fe55500a0a8\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") "
Oct 30 19:16:45.056381 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.056356    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-tmp-dir\") pod \"ff3d235d-adec-4667-9553-3fe55500a0a8\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") "
Oct 30 19:16:45.056428 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.056387    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-model-cache" pod "ff3d235d-adec-4667-9553-3fe55500a0a8" (UID: "ff3d235d-adec-4667-9553-3fe55500a0a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:45.056468 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.056438    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-home\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-home\") pod \"ff3d235d-adec-4667-9553-3fe55500a0a8\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") "
Oct 30 19:16:45.056529 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.056511    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-dshm\") pod \"ff3d235d-adec-4667-9553-3fe55500a0a8\" (UID: \"ff3d235d-adec-4667-9553-3fe55500a0a8\") "
Oct 30 19:16:45.056648 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.056622    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-tmp-dir" pod "ff3d235d-adec-4667-9553-3fe55500a0a8" (UID: "ff3d235d-adec-4667-9553-3fe55500a0a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:45.056797 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.056765    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-home" pod "ff3d235d-adec-4667-9553-3fe55500a0a8" (UID: "ff3d235d-adec-4667-9553-3fe55500a0a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:45.057126 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.057105    4391 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-model-cache\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.057200 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.057133    4391 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-tmp-dir\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.057200 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.057148    4391 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-home\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.058709 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.058678    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3d235d-adec-4667-9553-3fe55500a0a8-kube-api-access-rc7db" pod "ff3d235d-adec-4667-9553-3fe55500a0a8" (UID: "ff3d235d-adec-4667-9553-3fe55500a0a8"). InnerVolumeSpecName "kube-api-access-rc7db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:45.058791 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.058723    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-dshm" pod "ff3d235d-adec-4667-9553-3fe55500a0a8" (UID: "ff3d235d-adec-4667-9553-3fe55500a0a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:45.058791 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.058732    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff3d235d-adec-4667-9553-3fe55500a0a8-tls-certs" pod "ff3d235d-adec-4667-9553-3fe55500a0a8" (UID: "ff3d235d-adec-4667-9553-3fe55500a0a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:45.158240 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.158206    4391 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff3d235d-adec-4667-9553-3fe55500a0a8-dshm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.158240 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.158236    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc7db\" (UniqueName: \"kubernetes.io/projected/ff3d235d-adec-4667-9553-3fe55500a0a8-kube-api-access-rc7db\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.158240 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.158246    4391 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff3d235d-adec-4667-9553-3fe55500a0a8-tls-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.717373 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.717341    4391 generic.go:356] "Generic (PLEG): container finished" podID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerID="abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf" exitCode=0
Oct 30 19:16:45.717559 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.717383    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"
Oct 30 19:16:45.717559 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.717417    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" event={"ID":"ff3d235d-adec-4667-9553-3fe55500a0a8","Type":"ContainerDied","Data":"abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf"}
Oct 30 19:16:45.717559 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.717464    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h" event={"ID":"ff3d235d-adec-4667-9553-3fe55500a0a8","Type":"ContainerDied","Data":"ee794ea3a46b23d16fa234ea57827ca2ad3d56d8e889c1bc2f1ea1aace3a7412"}
Oct 30 19:16:45.717559 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.717484    4391 scope.go:122] "RemoveContainer" containerID="abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf"
Oct 30 19:16:45.727952 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.727936    4391 scope.go:122] "RemoveContainer" containerID="abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf"
Oct 30 19:16:45.728225 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:45.728208    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf\": container with ID starting with abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf not found: ID does not exist" containerID="abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf"
Oct 30 19:16:45.728267 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.728233    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf"} err="failed to get container status \"abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf\": rpc error: code = NotFound desc = could not find container \"abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf\": container with ID starting with abd5a59d855e29ce419486ad2b51c65c72dff01432a4152334126c9a69342ccf not found: ID does not exist"
Oct 30 19:16:45.875480 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.875447    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"]
Oct 30 19:16:45.921891 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:45.921847    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/premium-w4-75e91b-kserve-6b9958f8fc-h9s4h"]
Oct 30 19:16:47.005630 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:47.005570    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" path="/var/lib/kubelet/pods/ff3d235d-adec-4667-9553-3fe55500a0a8/volumes"
Oct 30 19:16:57.692546 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:57.692500    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main" containerID="cri-o://649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3" gracePeriod=45
Oct 30 19:16:57.958213 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:57.958156    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 19:16:58.016507 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.016479    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-kube-api-access-ldqdx\" (UniqueName: \"kubernetes.io/projected/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-kube-api-access-ldqdx\") pod \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") "
Oct 30 19:16:58.016674 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.016590    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tmp-dir\") pod \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") "
Oct 30 19:16:58.016674 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.016629    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-model-cache\") pod \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") "
Oct 30 19:16:58.016767 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.016740    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-home\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-home\") pod \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") "
Oct 30 19:16:58.016824 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.016771    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-dshm\") pod \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") "
Oct 30 19:16:58.016878 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.016827    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tmp-dir" pod "dd19633f-bc62-4fc1-9a33-f0e40a541c8b" (UID: "dd19633f-bc62-4fc1-9a33-f0e40a541c8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.016878 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.016841    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tls-certs\") pod \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\" (UID: \"dd19633f-bc62-4fc1-9a33-f0e40a541c8b\") "
Oct 30 19:16:58.016878 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.016861    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-model-cache" pod "dd19633f-bc62-4fc1-9a33-f0e40a541c8b" (UID: "dd19633f-bc62-4fc1-9a33-f0e40a541c8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.017056 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.017028    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-home" pod "dd19633f-bc62-4fc1-9a33-f0e40a541c8b" (UID: "dd19633f-bc62-4fc1-9a33-f0e40a541c8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.017326 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.017303    4391 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tmp-dir\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.017404 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.017331    4391 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-model-cache\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.017404 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.017347    4391 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-home\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.018877 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.018847    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-kube-api-access-ldqdx" pod "dd19633f-bc62-4fc1-9a33-f0e40a541c8b" (UID: "dd19633f-bc62-4fc1-9a33-f0e40a541c8b"). InnerVolumeSpecName "kube-api-access-ldqdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.018978 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.018908    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tls-certs" pod "dd19633f-bc62-4fc1-9a33-f0e40a541c8b" (UID: "dd19633f-bc62-4fc1-9a33-f0e40a541c8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:58.018978 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.018915    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-dshm" pod "dd19633f-bc62-4fc1-9a33-f0e40a541c8b" (UID: "dd19633f-bc62-4fc1-9a33-f0e40a541c8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.118917 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.118882    4391 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-dshm\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.118917 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.118918    4391 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-tls-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.119124 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.118933    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldqdx\" (UniqueName: \"kubernetes.io/projected/dd19633f-bc62-4fc1-9a33-f0e40a541c8b-kube-api-access-ldqdx\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.796595 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.796551    4391 generic.go:356] "Generic (PLEG): container finished" podID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerID="649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3" exitCode=0
Oct 30 19:16:58.797025 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.796621    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"
Oct 30 19:16:58.797025 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.796648    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" event={"ID":"dd19633f-bc62-4fc1-9a33-f0e40a541c8b","Type":"ContainerDied","Data":"649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3"}
Oct 30 19:16:58.797025 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.796696    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf" event={"ID":"dd19633f-bc62-4fc1-9a33-f0e40a541c8b","Type":"ContainerDied","Data":"7d961d411cb670c2ee655027e2fc7acad43f493cab564a9957b183bd98862a3e"}
Oct 30 19:16:58.797025 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.796718    4391 scope.go:122] "RemoveContainer" containerID="649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3"
Oct 30 19:16:58.811213 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.811184    4391 scope.go:122] "RemoveContainer" containerID="649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3"
Oct 30 19:16:58.811743 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:58.811715    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3\": container with ID starting with 649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3 not found: ID does not exist" containerID="649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3"
Oct 30 19:16:58.811837 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.811755    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3"} err="failed to get container status \"649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3\": rpc error: code = NotFound desc = could not find container \"649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3\": container with ID starting with 649fedf4cbe2c3bcbc4821f4b1811a5b8b515d62c9c74e81856fdf489f53e4d3 not found: ID does not exist"
Oct 30 19:16:58.955260 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:58.955221    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"]
Oct 30 19:16:59.005831 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.005801    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/premium-w2-ec8dab-kserve-6db555c58f-dwbpf"]
Oct 30 19:16:59.211119 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.211085    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"]
Oct 30 19:16:59.211395 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.211363    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf" podUID="b59a45ca-edd6-45e3-b55b-1a4456ab0701" containerName="maas-api" containerID="cri-o://4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02" gracePeriod=30
Oct 30 19:16:59.576344 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.576317    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 19:16:59.661604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.660910    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-metrics-certs\") pod \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") "
Oct 30 19:16:59.661604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.660980    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-maas-api-tls\") pod \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") "
Oct 30 19:16:59.661604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.661105    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b59a45ca-edd6-45e3-b55b-1a4456ab0701-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59a45ca-edd6-45e3-b55b-1a4456ab0701-openshift-service-ca\") pod \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") "
Oct 30 19:16:59.661604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.661202    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59a45ca-edd6-45e3-b55b-1a4456ab0701-kube-api-access-v569v\" (UniqueName: \"kubernetes.io/projected/b59a45ca-edd6-45e3-b55b-1a4456ab0701-kube-api-access-v569v\") pod \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\" (UID: \"b59a45ca-edd6-45e3-b55b-1a4456ab0701\") "
Oct 30 19:16:59.664305 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.662407    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59a45ca-edd6-45e3-b55b-1a4456ab0701-openshift-service-ca" pod "b59a45ca-edd6-45e3-b55b-1a4456ab0701" (UID: "b59a45ca-edd6-45e3-b55b-1a4456ab0701"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:59.681417 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.681170    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-metrics-certs" pod "b59a45ca-edd6-45e3-b55b-1a4456ab0701" (UID: "b59a45ca-edd6-45e3-b55b-1a4456ab0701"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:59.684735 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.683872    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59a45ca-edd6-45e3-b55b-1a4456ab0701-kube-api-access-v569v" pod "b59a45ca-edd6-45e3-b55b-1a4456ab0701" (UID: "b59a45ca-edd6-45e3-b55b-1a4456ab0701"). InnerVolumeSpecName "kube-api-access-v569v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:59.684735 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.684434    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-maas-api-tls" pod "b59a45ca-edd6-45e3-b55b-1a4456ab0701" (UID: "b59a45ca-edd6-45e3-b55b-1a4456ab0701"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:59.763224 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.763187    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:59.763224 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.763218    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b59a45ca-edd6-45e3-b55b-1a4456ab0701-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:59.763422 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.763232    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59a45ca-edd6-45e3-b55b-1a4456ab0701-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:59.763422 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.763246    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v569v\" (UniqueName: \"kubernetes.io/projected/b59a45ca-edd6-45e3-b55b-1a4456ab0701-kube-api-access-v569v\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:59.835665 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.835542    4391 generic.go:356] "Generic (PLEG): container finished" podID="b59a45ca-edd6-45e3-b55b-1a4456ab0701" containerID="4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02" exitCode=0
Oct 30 19:16:59.835665 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.835633    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf" event={"ID":"b59a45ca-edd6-45e3-b55b-1a4456ab0701","Type":"ContainerDied","Data":"4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02"}
Oct 30 19:16:59.835665 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.835653    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"
Oct 30 19:16:59.836304 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.835695    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf" event={"ID":"b59a45ca-edd6-45e3-b55b-1a4456ab0701","Type":"ContainerDied","Data":"fa41dbaf254f4cbbdbd181cf8f574317786869c775f7ff291dec22471ba2b7fb"}
Oct 30 19:16:59.836304 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.835745    4391 scope.go:122] "RemoveContainer" containerID="4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02"
Oct 30 19:16:59.850205 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.850187    4391 scope.go:122] "RemoveContainer" containerID="4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02"
Oct 30 19:16:59.850478 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:16:59.850460    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02\": container with ID starting with 4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02 not found: ID does not exist" containerID="4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02"
Oct 30 19:16:59.850541 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.850489    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02"} err="failed to get container status \"4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02\": rpc error: code = NotFound desc = could not find container \"4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02\": container with ID starting with 4bfcdbb612b66991b363e29648b091d4b655b4f262d3bb2dfad32751ff9bad02 not found: ID does not exist"
Oct 30 19:16:59.994503 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:16:59.994470    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"]
Oct 30 19:17:00.003076 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:00.003046    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a1617e-5768bbd67b-bzzbf"]
Oct 30 19:17:01.008227 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:01.008189    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b59a45ca-edd6-45e3-b55b-1a4456ab0701" path="/var/lib/kubelet/pods/b59a45ca-edd6-45e3-b55b-1a4456ab0701/volumes"
Oct 30 19:17:01.008727 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:01.008689    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" path="/var/lib/kubelet/pods/dd19633f-bc62-4fc1-9a33-f0e40a541c8b/volumes"
Oct 30 19:17:04.365440 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:04.365399    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"]
Oct 30 19:17:04.365944 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:04.365650    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerName="istio-proxy" containerID="cri-o://44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae" gracePeriod=30
Oct 30 19:17:04.389399 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:04.389359    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"]
Oct 30 19:17:04.389620 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:04.389562    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" podUID="c52d0536-fca2-49cf-b0af-9d78f5b14829" containerName="istio-proxy" containerID="cri-o://2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6" gracePeriod=30
Oct 30 19:17:09.368845 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.368806    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"]
Oct 30 19:17:09.369347 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.369063    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy" containerID="cri-o://a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1" gracePeriod=30
Oct 30 19:17:09.699700 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.699673    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:17:09.737514 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.737491    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:17:09.814525 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.814485    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-credential-socket\") pod \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") "
Oct 30 19:17:09.814741 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.814596    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-data\") pod \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") "
Oct 30 19:17:09.814741 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.814634    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istiod-ca-cert\") pod \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") "
Oct 30 19:17:09.814741 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.814684    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-kube-api-access-lqkxq\" (UniqueName: \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-kube-api-access-lqkxq\") pod \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") "
Oct 30 19:17:09.814741 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.814730    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-socket\") pod \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") "
Oct 30 19:17:09.814974 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.814775    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-envoy\") pod \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") "
Oct 30 19:17:09.814974 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.814828    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-credential-socket" pod "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" (UID: "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.814974 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.814843    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-certs\") pod \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") "
Oct 30 19:17:09.814974 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.814910    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-data" pod "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" (UID: "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.815389 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.814984    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-podinfo\") pod \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") "
Oct 30 19:17:09.815389 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.815018    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-token\" (UniqueName: \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-token\") pod \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\" (UID: \"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec\") "
Oct 30 19:17:09.815389 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.815056    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-certs" pod "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" (UID: "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.815389 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.815190    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-socket" pod "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" (UID: "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.816021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.815502    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.816021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.815524    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.816021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.815539    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.816021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.815552    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.816350 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.816321    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istiod-ca-cert" pod "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" (UID: "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:09.818015 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.817992    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-kube-api-access-lqkxq" pod "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" (UID: "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec"). InnerVolumeSpecName "kube-api-access-lqkxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.818127 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.818017    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-token" pod "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" (UID: "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.818127 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.818107    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-envoy" pod "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" (UID: "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.818791 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.818759    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-podinfo" pod "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" (UID: "334efd0f-aefb-4c27-baf9-f8e51ef0b6ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:09.906139 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.906104    4391 generic.go:356] "Generic (PLEG): container finished" podID="c52d0536-fca2-49cf-b0af-9d78f5b14829" containerID="2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6" exitCode=0
Oct 30 19:17:09.906345 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.906157    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"
Oct 30 19:17:09.906345 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.906193    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" event={"ID":"c52d0536-fca2-49cf-b0af-9d78f5b14829","Type":"ContainerDied","Data":"2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6"}
Oct 30 19:17:09.906345 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.906255    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h" event={"ID":"c52d0536-fca2-49cf-b0af-9d78f5b14829","Type":"ContainerDied","Data":"27d47626829c51a3a5fecfebde039dd69eb9b16f265bff546a40083aadf7c5a0"}
Oct 30 19:17:09.906345 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.906284    4391 scope.go:122] "RemoveContainer" containerID="2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6"
Oct 30 19:17:09.907816 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.907791    4391 generic.go:356] "Generic (PLEG): container finished" podID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerID="44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae" exitCode=0
Oct 30 19:17:09.907982 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.907961    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" event={"ID":"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec","Type":"ContainerDied","Data":"44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae"}
Oct 30 19:17:09.908077 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.908010    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p" event={"ID":"334efd0f-aefb-4c27-baf9-f8e51ef0b6ec","Type":"ContainerDied","Data":"33dfda68ec4e8f45f047cd6aa0e2e48cea3f2e7ec51fb54be97f0dbccca5f373"}
Oct 30 19:17:09.908143 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.908080    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"
Oct 30 19:17:09.916519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.916499    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-certs\") pod \"c52d0536-fca2-49cf-b0af-9d78f5b14829\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") "
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.916749    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-envoy\") pod \"c52d0536-fca2-49cf-b0af-9d78f5b14829\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") "
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.916802    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-credential-socket\") pod \"c52d0536-fca2-49cf-b0af-9d78f5b14829\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") "
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.916834    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-socket\") pod \"c52d0536-fca2-49cf-b0af-9d78f5b14829\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") "
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.916833    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-certs" pod "c52d0536-fca2-49cf-b0af-9d78f5b14829" (UID: "c52d0536-fca2-49cf-b0af-9d78f5b14829"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.916896    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-podinfo\") pod \"c52d0536-fca2-49cf-b0af-9d78f5b14829\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") "
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.916927    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-token\" (UniqueName: \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-token\") pod \"c52d0536-fca2-49cf-b0af-9d78f5b14829\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") "
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.916968    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52d0536-fca2-49cf-b0af-9d78f5b14829-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52d0536-fca2-49cf-b0af-9d78f5b14829-istiod-ca-cert\") pod \"c52d0536-fca2-49cf-b0af-9d78f5b14829\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") "
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.917002    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-data\") pod \"c52d0536-fca2-49cf-b0af-9d78f5b14829\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") "
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.917042    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-kube-api-access-kwhd6\" (UniqueName: \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-kube-api-access-kwhd6\") pod \"c52d0536-fca2-49cf-b0af-9d78f5b14829\" (UID: \"c52d0536-fca2-49cf-b0af-9d78f5b14829\") "
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.917326    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-socket" pod "c52d0536-fca2-49cf-b0af-9d78f5b14829" (UID: "c52d0536-fca2-49cf-b0af-9d78f5b14829"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.917414    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.917437    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqkxq\" (UniqueName: \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-kube-api-access-lqkxq\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.917479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.917453    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.918376 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.918351    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.918510 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.918378    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.918510 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.918396    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.918510 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.918411    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.918510 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.917410    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-credential-socket" pod "c52d0536-fca2-49cf-b0af-9d78f5b14829" (UID: "c52d0536-fca2-49cf-b0af-9d78f5b14829"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.918510 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.917647    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52d0536-fca2-49cf-b0af-9d78f5b14829-istiod-ca-cert" pod "c52d0536-fca2-49cf-b0af-9d78f5b14829" (UID: "c52d0536-fca2-49cf-b0af-9d78f5b14829"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:09.918510 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.917844    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-data" pod "c52d0536-fca2-49cf-b0af-9d78f5b14829" (UID: "c52d0536-fca2-49cf-b0af-9d78f5b14829"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.918510 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.917743    4391 scope.go:122] "RemoveContainer" containerID="2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6"
Oct 30 19:17:09.918928 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:17:09.918779    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6\": container with ID starting with 2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6 not found: ID does not exist" containerID="2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6"
Oct 30 19:17:09.918928 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.918814    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6"} err="failed to get container status \"2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6\": rpc error: code = NotFound desc = could not find container \"2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6\": container with ID starting with 2f7c9c831882c0f1db252933390b9d02f1755519d2b4d722143c33048d4ae3d6 not found: ID does not exist"
Oct 30 19:17:09.918928 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.918842    4391 scope.go:122] "RemoveContainer" containerID="44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae"
Oct 30 19:17:09.919187 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.919166    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-envoy" pod "c52d0536-fca2-49cf-b0af-9d78f5b14829" (UID: "c52d0536-fca2-49cf-b0af-9d78f5b14829"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.919766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.919740    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-podinfo" pod "c52d0536-fca2-49cf-b0af-9d78f5b14829" (UID: "c52d0536-fca2-49cf-b0af-9d78f5b14829"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:09.919861 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.919822    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-token" pod "c52d0536-fca2-49cf-b0af-9d78f5b14829" (UID: "c52d0536-fca2-49cf-b0af-9d78f5b14829"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.920291 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.920272    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-kube-api-access-kwhd6" pod "c52d0536-fca2-49cf-b0af-9d78f5b14829" (UID: "c52d0536-fca2-49cf-b0af-9d78f5b14829"). InnerVolumeSpecName "kube-api-access-kwhd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.928075 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.928039    4391 scope.go:122] "RemoveContainer" containerID="44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae"
Oct 30 19:17:09.928370 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:17:09.928344    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae\": container with ID starting with 44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae not found: ID does not exist" containerID="44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae"
Oct 30 19:17:09.928468 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:09.928379    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae"} err="failed to get container status \"44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae\": rpc error: code = NotFound desc = could not find container \"44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae\": container with ID starting with 44b3e92810653aeb1a8acf86475310fa53c97fb8b17f3536bfb46a03d78feeae not found: ID does not exist"
Oct 30 19:17:10.019749 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.019708    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:10.019749 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.019734    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52d0536-fca2-49cf-b0af-9d78f5b14829-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:10.019749 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.019743    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:10.019749 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.019752    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwhd6\" (UniqueName: \"kubernetes.io/projected/c52d0536-fca2-49cf-b0af-9d78f5b14829-kube-api-access-kwhd6\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:10.019749 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.019761    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:10.020225 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.019770    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d0536-fca2-49cf-b0af-9d78f5b14829-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:10.020225 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.019779    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52d0536-fca2-49cf-b0af-9d78f5b14829-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:10.036514 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.036479    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"]
Oct 30 19:17:10.044321 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.044267    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-n946p"]
Oct 30 19:17:10.323526 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.323488    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"]
Oct 30 19:17:10.326498 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:10.326469    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-mf65h"]
Oct 30 19:17:11.007897 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:11.007862    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" path="/var/lib/kubelet/pods/334efd0f-aefb-4c27-baf9-f8e51ef0b6ec/volumes"
Oct 30 19:17:11.008353 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:11.008339    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52d0536-fca2-49cf-b0af-9d78f5b14829" path="/var/lib/kubelet/pods/c52d0536-fca2-49cf-b0af-9d78f5b14829/volumes"
Oct 30 19:17:11.252558 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:17:11.252518    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:17:14.634873 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.634849    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 19:17:14.667519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.667498    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-envoy\") pod \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") "
Oct 30 19:17:14.667652 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.667554    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-credential-socket\") pod \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") "
Oct 30 19:17:14.667652 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.667612    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-certs\") pod \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") "
Oct 30 19:17:14.667752 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.667654    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-token\" (UniqueName: \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-token\") pod \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") "
Oct 30 19:17:14.667752 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.667703    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-data\") pod \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") "
Oct 30 19:17:14.667752 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.667724    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-kube-api-access-7kwvr\" (UniqueName: \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-kube-api-access-7kwvr\") pod \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") "
Oct 30 19:17:14.667910 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.667749    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-podinfo\") pod \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") "
Oct 30 19:17:14.667910 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.667789    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istiod-ca-cert\") pod \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") "
Oct 30 19:17:14.667910 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.667850    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-socket\") pod \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\" (UID: \"483f0f89-fa50-4e5a-a901-60d7cf6bb333\") "
Oct 30 19:17:14.668194 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.668165    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-credential-socket" pod "483f0f89-fa50-4e5a-a901-60d7cf6bb333" (UID: "483f0f89-fa50-4e5a-a901-60d7cf6bb333"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.668282 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.668209    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-certs" pod "483f0f89-fa50-4e5a-a901-60d7cf6bb333" (UID: "483f0f89-fa50-4e5a-a901-60d7cf6bb333"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.668345 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.668275    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-data" pod "483f0f89-fa50-4e5a-a901-60d7cf6bb333" (UID: "483f0f89-fa50-4e5a-a901-60d7cf6bb333"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.668457 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.668428    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-socket" pod "483f0f89-fa50-4e5a-a901-60d7cf6bb333" (UID: "483f0f89-fa50-4e5a-a901-60d7cf6bb333"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.668618 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.668594    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istiod-ca-cert" pod "483f0f89-fa50-4e5a-a901-60d7cf6bb333" (UID: "483f0f89-fa50-4e5a-a901-60d7cf6bb333"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:14.669824 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.669796    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-envoy" pod "483f0f89-fa50-4e5a-a901-60d7cf6bb333" (UID: "483f0f89-fa50-4e5a-a901-60d7cf6bb333"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.669824 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.669817    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-token" pod "483f0f89-fa50-4e5a-a901-60d7cf6bb333" (UID: "483f0f89-fa50-4e5a-a901-60d7cf6bb333"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.669959 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.669855    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-kube-api-access-7kwvr" pod "483f0f89-fa50-4e5a-a901-60d7cf6bb333" (UID: "483f0f89-fa50-4e5a-a901-60d7cf6bb333"). InnerVolumeSpecName "kube-api-access-7kwvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.669959 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.669868    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-podinfo" pod "483f0f89-fa50-4e5a-a901-60d7cf6bb333" (UID: "483f0f89-fa50-4e5a-a901-60d7cf6bb333"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:14.769283 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.769248    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.769283 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.769276    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.769283 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.769287    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kwvr\" (UniqueName: \"kubernetes.io/projected/483f0f89-fa50-4e5a-a901-60d7cf6bb333-kube-api-access-7kwvr\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.769519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.769297    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.769519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.769307    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.769519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.769316    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.769519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.769324    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.769519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.769332    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.769519 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.769340    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/483f0f89-fa50-4e5a-a901-60d7cf6bb333-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.941952 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.941851    4391 generic.go:356] "Generic (PLEG): container finished" podID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerID="a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1" exitCode=0
Oct 30 19:17:14.941952 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.941904    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"
Oct 30 19:17:14.941952 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.941940    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" event={"ID":"483f0f89-fa50-4e5a-a901-60d7cf6bb333","Type":"ContainerDied","Data":"a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1"}
Oct 30 19:17:14.942228 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.941995    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s" event={"ID":"483f0f89-fa50-4e5a-a901-60d7cf6bb333","Type":"ContainerDied","Data":"e9e499038f38ed922b7448971fcce95c49d140fc483255a19a2fd11aaf0b8c33"}
Oct 30 19:17:14.942228 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.942018    4391 scope.go:122] "RemoveContainer" containerID="a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1"
Oct 30 19:17:14.952677 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.952656    4391 scope.go:122] "RemoveContainer" containerID="a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1"
Oct 30 19:17:14.952954 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:17:14.952935    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1\": container with ID starting with a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1 not found: ID does not exist" containerID="a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1"
Oct 30 19:17:14.953004 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:14.952962    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1"} err="failed to get container status \"a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1\": rpc error: code = NotFound desc = could not find container \"a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1\": container with ID starting with a73efab8457228c7edd8b35a4bdb7963f17925511ade28525eba5b3ce34840b1 not found: ID does not exist"
Oct 30 19:17:15.102786 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:15.102723    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"]
Oct 30 19:17:15.109312 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:15.109282    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-68t8s"]
Oct 30 19:17:15.770731 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:15.770696    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:17:17.006495 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:17.006451    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" path="/var/lib/kubelet/pods/483f0f89-fa50-4e5a-a901-60d7cf6bb333/volumes"
Oct 30 19:17:26.007341 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.005718    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"]
Oct 30 19:17:26.007341 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.006019    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc" podUID="97483113-cab0-47f1-9ded-0f2814455361" containerName="maas-api" containerID="cri-o://99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21" gracePeriod=30
Oct 30 19:17:26.302748 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.302722    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 19:17:26.403605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.403177    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-metrics-certs\") pod \"97483113-cab0-47f1-9ded-0f2814455361\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") "
Oct 30 19:17:26.403605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.403237    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97483113-cab0-47f1-9ded-0f2814455361-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97483113-cab0-47f1-9ded-0f2814455361-openshift-service-ca\") pod \"97483113-cab0-47f1-9ded-0f2814455361\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") "
Oct 30 19:17:26.403605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.403282    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-maas-api-tls\") pod \"97483113-cab0-47f1-9ded-0f2814455361\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") "
Oct 30 19:17:26.403605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.403352    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97483113-cab0-47f1-9ded-0f2814455361-kube-api-access-8f58h\" (UniqueName: \"kubernetes.io/projected/97483113-cab0-47f1-9ded-0f2814455361-kube-api-access-8f58h\") pod \"97483113-cab0-47f1-9ded-0f2814455361\" (UID: \"97483113-cab0-47f1-9ded-0f2814455361\") "
Oct 30 19:17:26.408603 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.405267    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97483113-cab0-47f1-9ded-0f2814455361-openshift-service-ca" pod "97483113-cab0-47f1-9ded-0f2814455361" (UID: "97483113-cab0-47f1-9ded-0f2814455361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:26.426385 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.421046    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-maas-api-tls" pod "97483113-cab0-47f1-9ded-0f2814455361" (UID: "97483113-cab0-47f1-9ded-0f2814455361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:26.426780 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.426732    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97483113-cab0-47f1-9ded-0f2814455361-kube-api-access-8f58h" pod "97483113-cab0-47f1-9ded-0f2814455361" (UID: "97483113-cab0-47f1-9ded-0f2814455361"). InnerVolumeSpecName "kube-api-access-8f58h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:26.430089 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.430054    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-metrics-certs" pod "97483113-cab0-47f1-9ded-0f2814455361" (UID: "97483113-cab0-47f1-9ded-0f2814455361"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:26.504830 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.504795    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:26.504830 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.504825    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97483113-cab0-47f1-9ded-0f2814455361-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:26.504830 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.504834    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97483113-cab0-47f1-9ded-0f2814455361-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:26.505052 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:26.504843    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f58h\" (UniqueName: \"kubernetes.io/projected/97483113-cab0-47f1-9ded-0f2814455361-kube-api-access-8f58h\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:27.077161 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:27.077123    4391 generic.go:356] "Generic (PLEG): container finished" podID="97483113-cab0-47f1-9ded-0f2814455361" containerID="99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21" exitCode=0
Oct 30 19:17:27.078900 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:27.077241    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc" event={"ID":"97483113-cab0-47f1-9ded-0f2814455361","Type":"ContainerDied","Data":"99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21"}
Oct 30 19:17:27.078900 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:27.077297    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc" event={"ID":"97483113-cab0-47f1-9ded-0f2814455361","Type":"ContainerDied","Data":"d27a51e8ec7e9fedd7c535a5bff00e5e893ebc4f64ba011bee8ba77582f1c17e"}
Oct 30 19:17:27.078900 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:27.077327    4391 scope.go:122] "RemoveContainer" containerID="99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21"
Oct 30 19:17:27.078900 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:27.077497    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"
Oct 30 19:17:27.090778 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:27.090749    4391 scope.go:122] "RemoveContainer" containerID="99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21"
Oct 30 19:17:27.091114 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:17:27.091091    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21\": container with ID starting with 99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21 not found: ID does not exist" containerID="99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21"
Oct 30 19:17:27.091178 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:27.091125    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21"} err="failed to get container status \"99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21\": rpc error: code = NotFound desc = could not find container \"99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21\": container with ID starting with 99b03e04ab058849ae155f18ad82eab0e2a1fa1069f04e19d300661a232d6d21 not found: ID does not exist"
Oct 30 19:17:27.416731 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:27.416694    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"]
Oct 30 19:17:27.588556 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:27.588514    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-75e91b-6997bd8f85-w74kc"]
Oct 30 19:17:29.005424 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:29.005389    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97483113-cab0-47f1-9ded-0f2814455361" path="/var/lib/kubelet/pods/97483113-cab0-47f1-9ded-0f2814455361/volumes"
Oct 30 19:17:36.303522 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:36.303490    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"]
Oct 30 19:17:36.304043 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:36.303769    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy" containerID="cri-o://3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba" gracePeriod=30
Oct 30 19:17:36.306169 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:36.306142    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"]
Oct 30 19:17:36.306332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:36.306314    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy" containerID="cri-o://44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3" gracePeriod=30
Oct 30 19:17:37.428475 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:37.428431    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"]
Oct 30 19:17:37.429006 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:37.428752    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy" containerID="cri-o://25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9" gracePeriod=30
Oct 30 19:17:41.269451 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:17:41.269399    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:17:41.565559 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.565532    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 19:17:41.599173 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.599150    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 19:17:41.687423 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687385    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-certs\") pod \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") "
Oct 30 19:17:41.687660 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687430    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-socket\") pod \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") "
Oct 30 19:17:41.687660 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687461    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-podinfo\") pod \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") "
Oct 30 19:17:41.687660 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687490    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-podinfo\") pod \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") "
Oct 30 19:17:41.687660 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687518    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-credential-socket\") pod \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") "
Oct 30 19:17:41.687660 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687572    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-envoy\") pod \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") "
Oct 30 19:17:41.687953 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687705    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-token\" (UniqueName: \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-token\") pod \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") "
Oct 30 19:17:41.687953 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687718    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-certs" pod "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" (UID: "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.687953 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687742    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-socket" pod "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" (UID: "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.687953 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687755    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istiod-ca-cert\") pod \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") "
Oct 30 19:17:41.687953 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687788    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-token\" (UniqueName: \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-token\") pod \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") "
Oct 30 19:17:41.687953 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687817    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-data\") pod \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") "
Oct 30 19:17:41.687953 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687844    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istiod-ca-cert\") pod \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") "
Oct 30 19:17:41.687953 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.687898    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-credential-socket" pod "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" (UID: "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.688364 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688179    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istiod-ca-cert" pod "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" (UID: "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:41.688364 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688210    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istiod-ca-cert" pod "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" (UID: "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:41.688364 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688227    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-credential-socket\") pod \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") "
Oct 30 19:17:41.688364 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688271    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-kube-api-access-fs7xx\" (UniqueName: \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-kube-api-access-fs7xx\") pod \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") "
Oct 30 19:17:41.688364 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688303    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-socket\") pod \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") "
Oct 30 19:17:41.688652 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688464    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-kube-api-access-8zjgh\" (UniqueName: \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-kube-api-access-8zjgh\") pod \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") "
Oct 30 19:17:41.688652 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688498    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-data" pod "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" (UID: "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.688652 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688524    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-data\") pod \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") "
Oct 30 19:17:41.688652 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688622    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-envoy\") pod \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\" (UID: \"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37\") "
Oct 30 19:17:41.688878 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688668    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-certs\") pod \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\" (UID: \"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d\") "
Oct 30 19:17:41.688878 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.688792    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-credential-socket" pod "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" (UID: "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.689236 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.689211    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.689342 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.689241    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.689342 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.689257    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.689342 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.689272    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.689342 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.689286    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.689342 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.689299    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.689342 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.689313    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.689710 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.689513    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-certs" pod "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" (UID: "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.689773 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.689716    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-data" pod "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" (UID: "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.690379 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.690347    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-envoy" pod "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" (UID: "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.690488 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.690426    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-podinfo" pod "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" (UID: "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:41.690550 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.690516    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-token" pod "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" (UID: "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:41.690550 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.690515    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-podinfo" pod "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" (UID: "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:41.690691 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.690671    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-socket" pod "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" (UID: "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.690940 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.690914    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-kube-api-access-fs7xx" pod "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" (UID: "8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d"). InnerVolumeSpecName "kube-api-access-fs7xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:41.691050 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.690967    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-token" pod "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" (UID: "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:41.691609 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.691563    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-kube-api-access-8zjgh" pod "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" (UID: "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37"). InnerVolumeSpecName "kube-api-access-8zjgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:41.691733 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.691717    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-envoy" pod "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" (UID: "a7516c72-37c4-4f4a-aeb6-eb7eb0baee37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.790331 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790252    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.790331 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790291    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.790331 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790308    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs7xx\" (UniqueName: \"kubernetes.io/projected/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-kube-api-access-fs7xx\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.790331 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790324    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.790631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790338    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zjgh\" (UniqueName: \"kubernetes.io/projected/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-kube-api-access-8zjgh\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.790631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790352    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.790631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790366    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.790631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790381    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.790631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790395    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.790631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790412    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.790631 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:41.790425    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.169237 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.169199    4391 generic.go:356] "Generic (PLEG): container finished" podID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerID="44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3" exitCode=0
Oct 30 19:17:42.169445 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.169253    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"
Oct 30 19:17:42.169445 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.169249    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" event={"ID":"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d","Type":"ContainerDied","Data":"44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3"}
Oct 30 19:17:42.169445 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.169375    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z" event={"ID":"8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d","Type":"ContainerDied","Data":"e7f6f9afb94307f3c3720dd10dee6aadca003ca7d674428a2af4bff88d13b9b5"}
Oct 30 19:17:42.169445 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.169402    4391 scope.go:122] "RemoveContainer" containerID="44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3"
Oct 30 19:17:42.171161 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.170983    4391 generic.go:356] "Generic (PLEG): container finished" podID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerID="3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba" exitCode=0
Oct 30 19:17:42.171161 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.171053    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"
Oct 30 19:17:42.171161 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.171085    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" event={"ID":"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37","Type":"ContainerDied","Data":"3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba"}
Oct 30 19:17:42.171161 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.171124    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp" event={"ID":"a7516c72-37c4-4f4a-aeb6-eb7eb0baee37","Type":"ContainerDied","Data":"b46fdaa02d1a65d6e664b91c1dfb9d2b0515a345547d18e45756d2b0c5f15730"}
Oct 30 19:17:42.179344 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.179329    4391 scope.go:122] "RemoveContainer" containerID="44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3"
Oct 30 19:17:42.179719 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:17:42.179692    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3\": container with ID starting with 44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3 not found: ID does not exist" containerID="44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3"
Oct 30 19:17:42.179791 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.179729    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3"} err="failed to get container status \"44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3\": rpc error: code = NotFound desc = could not find container \"44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3\": container with ID starting with 44604de632d3e220fa72ed4fd68ffd13987014d3dd01a32f8d54e3e6f327e8a3 not found: ID does not exist"
Oct 30 19:17:42.179791 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.179753    4391 scope.go:122] "RemoveContainer" containerID="3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba"
Oct 30 19:17:42.188424 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.188404    4391 scope.go:122] "RemoveContainer" containerID="3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba"
Oct 30 19:17:42.188696 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:17:42.188678    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba\": container with ID starting with 3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba not found: ID does not exist" containerID="3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba"
Oct 30 19:17:42.188768 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.188707    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba"} err="failed to get container status \"3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba\": rpc error: code = NotFound desc = could not find container \"3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba\": container with ID starting with 3c6b06b724f33e1be24ee7e03c3eba2b4cf416fc63e8991068424622a60593ba not found: ID does not exist"
Oct 30 19:17:42.331248 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.331209    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"]
Oct 30 19:17:42.377976 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.377947    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-h6w8z"]
Oct 30 19:17:42.448672 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.448572    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"]
Oct 30 19:17:42.455200 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.455168    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-xgjzp"]
Oct 30 19:17:42.700714 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.700649    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 19:17:42.798688 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.798653    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-credential-socket\") pod \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") "
Oct 30 19:17:42.798864 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.798741    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-token\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-token\") pod \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") "
Oct 30 19:17:42.798864 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.798786    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-socket\") pod \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") "
Oct 30 19:17:42.798864 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.798837    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istiod-ca-cert\") pod \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") "
Oct 30 19:17:42.798998 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.798906    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-data\") pod \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") "
Oct 30 19:17:42.798998 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.798940    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-envoy\") pod \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") "
Oct 30 19:17:42.798998 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.798967    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-credential-socket" pod "5fb5b8ff-d0e5-4e44-943a-35eeee70716f" (UID: "5fb5b8ff-d0e5-4e44-943a-35eeee70716f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.798998 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.798990    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-certs\") pod \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") "
Oct 30 19:17:42.799220 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799037    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-podinfo\") pod \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") "
Oct 30 19:17:42.799220 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799082    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-kube-api-access-t4nhf\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-kube-api-access-t4nhf\") pod \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\" (UID: \"5fb5b8ff-d0e5-4e44-943a-35eeee70716f\") "
Oct 30 19:17:42.799220 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799153    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-socket" pod "5fb5b8ff-d0e5-4e44-943a-35eeee70716f" (UID: "5fb5b8ff-d0e5-4e44-943a-35eeee70716f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.799373 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799241    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-data" pod "5fb5b8ff-d0e5-4e44-943a-35eeee70716f" (UID: "5fb5b8ff-d0e5-4e44-943a-35eeee70716f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.799373 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799262    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istiod-ca-cert" pod "5fb5b8ff-d0e5-4e44-943a-35eeee70716f" (UID: "5fb5b8ff-d0e5-4e44-943a-35eeee70716f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:42.799469 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799429    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-certs" pod "5fb5b8ff-d0e5-4e44-943a-35eeee70716f" (UID: "5fb5b8ff-d0e5-4e44-943a-35eeee70716f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.799596 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799525    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.799596 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799549    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.799596 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799564    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.799596 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799592    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.799740 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.799607    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.801236 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.801206    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-podinfo" pod "5fb5b8ff-d0e5-4e44-943a-35eeee70716f" (UID: "5fb5b8ff-d0e5-4e44-943a-35eeee70716f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:42.801460 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.801439    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-envoy" pod "5fb5b8ff-d0e5-4e44-943a-35eeee70716f" (UID: "5fb5b8ff-d0e5-4e44-943a-35eeee70716f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.801558 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.801485    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-token" pod "5fb5b8ff-d0e5-4e44-943a-35eeee70716f" (UID: "5fb5b8ff-d0e5-4e44-943a-35eeee70716f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:42.801640 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.801623    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-kube-api-access-t4nhf" pod "5fb5b8ff-d0e5-4e44-943a-35eeee70716f" (UID: "5fb5b8ff-d0e5-4e44-943a-35eeee70716f"). InnerVolumeSpecName "kube-api-access-t4nhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:42.900396 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.900364    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.900396 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.900392    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.900396 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.900403    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.900647 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:42.900413    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4nhf\" (UniqueName: \"kubernetes.io/projected/5fb5b8ff-d0e5-4e44-943a-35eeee70716f-kube-api-access-t4nhf\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:43.006607 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.006503    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" path="/var/lib/kubelet/pods/8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d/volumes"
Oct 30 19:17:43.007045 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.007029    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" path="/var/lib/kubelet/pods/a7516c72-37c4-4f4a-aeb6-eb7eb0baee37/volumes"
Oct 30 19:17:43.178286 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.178252    4391 generic.go:356] "Generic (PLEG): container finished" podID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerID="25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9" exitCode=0
Oct 30 19:17:43.178441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.178324    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"
Oct 30 19:17:43.178441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.178336    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" event={"ID":"5fb5b8ff-d0e5-4e44-943a-35eeee70716f","Type":"ContainerDied","Data":"25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9"}
Oct 30 19:17:43.178441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.178393    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls" event={"ID":"5fb5b8ff-d0e5-4e44-943a-35eeee70716f","Type":"ContainerDied","Data":"24980cfff61365a9d1ea9f1cb125ae712bf2c11ce53c45e8c88f2fe70bf50fc5"}
Oct 30 19:17:43.178441 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.178423    4391 scope.go:122] "RemoveContainer" containerID="25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9"
Oct 30 19:17:43.194041 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.194019    4391 scope.go:122] "RemoveContainer" containerID="25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9"
Oct 30 19:17:43.194354 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:17:43.194332    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9\": container with ID starting with 25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9 not found: ID does not exist" containerID="25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9"
Oct 30 19:17:43.194437 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.194367    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9"} err="failed to get container status \"25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9\": rpc error: code = NotFound desc = could not find container \"25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9\": container with ID starting with 25ef6a38dec3fdac7d4efab828e47c2737379f1faa93481a4b8cfab6778d3aa9 not found: ID does not exist"
Oct 30 19:17:43.341332 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.341240    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"]
Oct 30 19:17:43.387948 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:43.387906    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-rnrls"]
Oct 30 19:17:45.005479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:45.005446    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" path="/var/lib/kubelet/pods/5fb5b8ff-d0e5-4e44-943a-35eeee70716f/volumes"
Oct 30 19:17:47.566143 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:17:47.566107    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:18:10.060658 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:18:10.060557    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:18:11.252488 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:18:11.252445    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:18:34.698144 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:18:34.698095    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:18:41.252994 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:18:41.252949    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:18:54.950191 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:18:54.950152    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:19:11.253204 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:19:11.253166    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:19:37.776420 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:19:37.776338    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:19:41.253309 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:19:41.253269    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:19:56.751406 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:19:56.751372    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:20:11.252111 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:20:11.252069    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:20:22.765164 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:22.765119    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:20:41.251973 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:20:41.251939    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:20:48.737143 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:48.737099    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"]
Oct 30 19:20:48.737675 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:48.737428    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69" podUID="61b5903c-5d65-4bd8-9018-5f0930e14260" containerName="maas-api" containerID="cri-o://7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be" gracePeriod=30
Oct 30 19:20:49.020087 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.020062    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:20:49.147945 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.147910    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b5903c-5d65-4bd8-9018-5f0930e14260-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5903c-5d65-4bd8-9018-5f0930e14260-openshift-service-ca\") pod \"61b5903c-5d65-4bd8-9018-5f0930e14260\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") "
Oct 30 19:20:49.148116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.147961    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b5903c-5d65-4bd8-9018-5f0930e14260-kube-api-access-7xwnx\" (UniqueName: \"kubernetes.io/projected/61b5903c-5d65-4bd8-9018-5f0930e14260-kube-api-access-7xwnx\") pod \"61b5903c-5d65-4bd8-9018-5f0930e14260\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") "
Oct 30 19:20:49.148116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.148047    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-maas-api-tls\") pod \"61b5903c-5d65-4bd8-9018-5f0930e14260\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") "
Oct 30 19:20:49.148116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.148096    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-metrics-certs\") pod \"61b5903c-5d65-4bd8-9018-5f0930e14260\" (UID: \"61b5903c-5d65-4bd8-9018-5f0930e14260\") "
Oct 30 19:20:49.148352 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.148324    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b5903c-5d65-4bd8-9018-5f0930e14260-openshift-service-ca" pod "61b5903c-5d65-4bd8-9018-5f0930e14260" (UID: "61b5903c-5d65-4bd8-9018-5f0930e14260"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:49.148622 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.148562    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5903c-5d65-4bd8-9018-5f0930e14260-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.150661 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.150632    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-maas-api-tls" pod "61b5903c-5d65-4bd8-9018-5f0930e14260" (UID: "61b5903c-5d65-4bd8-9018-5f0930e14260"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:49.150770 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.150665    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b5903c-5d65-4bd8-9018-5f0930e14260-kube-api-access-7xwnx" pod "61b5903c-5d65-4bd8-9018-5f0930e14260" (UID: "61b5903c-5d65-4bd8-9018-5f0930e14260"). InnerVolumeSpecName "kube-api-access-7xwnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:49.150770 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.150720    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-metrics-certs" pod "61b5903c-5d65-4bd8-9018-5f0930e14260" (UID: "61b5903c-5d65-4bd8-9018-5f0930e14260"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:49.198412 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.198377    4391 generic.go:356] "Generic (PLEG): container finished" podID="61b5903c-5d65-4bd8-9018-5f0930e14260" containerID="7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be" exitCode=0
Oct 30 19:20:49.198614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.198426    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"
Oct 30 19:20:49.198614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.198460    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69" event={"ID":"61b5903c-5d65-4bd8-9018-5f0930e14260","Type":"ContainerDied","Data":"7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be"}
Oct 30 19:20:49.198614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.198527    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69" event={"ID":"61b5903c-5d65-4bd8-9018-5f0930e14260","Type":"ContainerDied","Data":"77638d00ea28d00bc5371b1415a8d11a70d272ccfb19c728e7ca6bd1b1816174"}
Oct 30 19:20:49.198614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.198555    4391 scope.go:122] "RemoveContainer" containerID="7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be"
Oct 30 19:20:49.208382 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.208365    4391 scope.go:122] "RemoveContainer" containerID="7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be"
Oct 30 19:20:49.208648 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:20:49.208626    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be\": container with ID starting with 7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be not found: ID does not exist" containerID="7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be"
Oct 30 19:20:49.208723 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.208660    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be"} err="failed to get container status \"7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be\": rpc error: code = NotFound desc = could not find container \"7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be\": container with ID starting with 7d4bc224c1d0fbff25958e2aa92f4a1b97e8f139ef1aa75dd551586aa4c082be not found: ID does not exist"
Oct 30 19:20:49.253075 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.252870    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xwnx\" (UniqueName: \"kubernetes.io/projected/61b5903c-5d65-4bd8-9018-5f0930e14260-kube-api-access-7xwnx\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.253075 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.252914    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.253075 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.252931    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61b5903c-5d65-4bd8-9018-5f0930e14260-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.368823 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.368781    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"]
Oct 30 19:20:49.380155 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:49.380124    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-29de7b-6bd57948d5-8sd69"]
Oct 30 19:20:51.005719 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:51.005688    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b5903c-5d65-4bd8-9018-5f0930e14260" path="/var/lib/kubelet/pods/61b5903c-5d65-4bd8-9018-5f0930e14260/volumes"
Oct 30 19:20:53.942224 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:20:53.942190    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:21:10.550565 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.550481    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"]
Oct 30 19:21:10.551096 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551075    4391 cpu_manager.go:414] "containerMap: removing container" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main"
Oct 30 19:21:10.551193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551098    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main"
Oct 30 19:21:10.551193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551118    4391 cpu_manager.go:414] "containerMap: removing container" podUID="97483113-cab0-47f1-9ded-0f2814455361" containerName="maas-api"
Oct 30 19:21:10.551193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551124    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97483113-cab0-47f1-9ded-0f2814455361" containerName="maas-api"
Oct 30 19:21:10.551193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551138    4391 cpu_manager.go:414] "containerMap: removing container" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main"
Oct 30 19:21:10.551193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551145    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main"
Oct 30 19:21:10.551193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551154    4391 cpu_manager.go:414] "containerMap: removing container" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerName="cleanup"
Oct 30 19:21:10.551193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551159    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerName="cleanup"
Oct 30 19:21:10.551193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551167    4391 cpu_manager.go:414] "containerMap: removing container" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerName="istio-proxy"
Oct 30 19:21:10.551193 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551173    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551207    4391 cpu_manager.go:414] "containerMap: removing container" podUID="84a719e5-b8cd-421a-9869-83b39f95391f" containerName="cleanup"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551213    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a719e5-b8cd-421a-9869-83b39f95391f" containerName="cleanup"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551223    4391 cpu_manager.go:414] "containerMap: removing container" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551228    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551236    4391 cpu_manager.go:414] "containerMap: removing container" podUID="c7ce2533-9241-4180-9675-fcbd391c3105" containerName="maas-api"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551241    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ce2533-9241-4180-9675-fcbd391c3105" containerName="maas-api"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551247    4391 cpu_manager.go:414] "containerMap: removing container" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerName="cleanup"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551251    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca127f30-3020-42f7-af2d-d23ca6c79fcb" containerName="cleanup"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551257    4391 cpu_manager.go:414] "containerMap: removing container" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551261    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551270    4391 cpu_manager.go:414] "containerMap: removing container" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551274    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551281    4391 cpu_manager.go:414] "containerMap: removing container" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551327    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551339    4391 cpu_manager.go:414] "containerMap: removing container" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551346    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551357    4391 cpu_manager.go:414] "containerMap: removing container" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551362    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551369    4391 cpu_manager.go:414] "containerMap: removing container" podUID="c52d0536-fca2-49cf-b0af-9d78f5b14829" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551373    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52d0536-fca2-49cf-b0af-9d78f5b14829" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551382    4391 cpu_manager.go:414] "containerMap: removing container" podUID="b59a45ca-edd6-45e3-b55b-1a4456ab0701" containerName="maas-api"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551390    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b59a45ca-edd6-45e3-b55b-1a4456ab0701" containerName="maas-api"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551408    4391 cpu_manager.go:414] "containerMap: removing container" podUID="61b5903c-5d65-4bd8-9018-5f0930e14260" containerName="maas-api"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551415    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b5903c-5d65-4bd8-9018-5f0930e14260" containerName="maas-api"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551517    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="8651aaf5-b5f7-4bdd-8f00-c1c4e6bda25d" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551526    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ce2533-9241-4180-9675-fcbd391c3105" containerName="maas-api"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551535    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5514f78-b153-49dc-9d09-9ba5e104ce78" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551548    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="483f0f89-fa50-4e5a-a901-60d7cf6bb333" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551555    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7516c72-37c4-4f4a-aeb6-eb7eb0baee37" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551561    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb5b8ff-d0e5-4e44-943a-35eeee70716f" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551567    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85c4e22-99d6-4616-b2b1-3ce5bd0a8261" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551575    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52d0536-fca2-49cf-b0af-9d78f5b14829" containerName="istio-proxy"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551610    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="b59a45ca-edd6-45e3-b55b-1a4456ab0701" containerName="maas-api"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551623    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b5903c-5d65-4bd8-9018-5f0930e14260" containerName="maas-api"
Oct 30 19:21:10.551614 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551632    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3d235d-adec-4667-9553-3fe55500a0a8" containerName="main"
Oct 30 19:21:10.553439 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551642    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="97483113-cab0-47f1-9ded-0f2814455361" containerName="maas-api"
Oct 30 19:21:10.553439 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551649    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd19633f-bc62-4fc1-9a33-f0e40a541c8b" containerName="main"
Oct 30 19:21:10.553439 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551658    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f03dc4b-361e-419b-a1ae-46a2146e0e98" containerName="cleanup"
Oct 30 19:21:10.553439 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551667    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="334efd0f-aefb-4c27-baf9-f8e51ef0b6ec" containerName="istio-proxy"
Oct 30 19:21:10.553439 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.551676    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a719e5-b8cd-421a-9869-83b39f95391f" containerName="cleanup"
Oct 30 19:21:10.556702 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.556681    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:10.614479 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.614440    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"]
Oct 30 19:21:10.653028 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.652997    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:21:10.653028 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.653002    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:21:10.653028 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.653001    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:10.653261 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.653044    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:21:10.661021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.660998    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be07cf67-brn82\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:10.661144 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.661043    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgxg\" (UniqueName: \"kubernetes.io/projected/5ad27024-eaea-4355-8691-2ce6018b4c89-kube-api-access-gmgxg\") pod \"maas-api-revoke-sub-be07cf67-brn82\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:10.661254 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.661229    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be07cf67-brn82\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:10.762362 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.762323    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be07cf67-brn82\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:10.762575 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.762378    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgxg\" (UniqueName: \"kubernetes.io/projected/5ad27024-eaea-4355-8691-2ce6018b4c89-kube-api-access-gmgxg\") pod \"maas-api-revoke-sub-be07cf67-brn82\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:10.762575 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.762441    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be07cf67-brn82\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:10.763149 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.763130    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be07cf67-brn82\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:10.764874 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.764852    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be07cf67-brn82\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:10.818054 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.817987    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgxg\" (UniqueName: \"kubernetes.io/projected/5ad27024-eaea-4355-8691-2ce6018b4c89-kube-api-access-gmgxg\") pod \"maas-api-revoke-sub-be07cf67-brn82\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:10.870201 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:10.870168    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:11.027026 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:11.026997    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"]
Oct 30 19:21:11.244508 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:11.244481    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"]
Oct 30 19:21:11.247166 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:21:11.247134    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad27024_eaea_4355_8691_2ce6018b4c89.slice/crio-e8b0b980c00ea14eb698ecd1610532652713eef48ae21ed67cce874d4c64773e WatchSource:0}: Error finding container e8b0b980c00ea14eb698ecd1610532652713eef48ae21ed67cce874d4c64773e: Status 404 returned error can't find the container with id e8b0b980c00ea14eb698ecd1610532652713eef48ae21ed67cce874d4c64773e
Oct 30 19:21:11.255395 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:21:11.255369    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:21:11.344025 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:11.343923    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82" event={"ID":"5ad27024-eaea-4355-8691-2ce6018b4c89","Type":"ContainerStarted","Data":"e8b0b980c00ea14eb698ecd1610532652713eef48ae21ed67cce874d4c64773e"}
Oct 30 19:21:12.352377 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:12.352341    4391 generic.go:356] "Generic (PLEG): container finished" podID="5ad27024-eaea-4355-8691-2ce6018b4c89" containerID="4ad42dc321e0c19adf0392373efd3c5cefbe0dbd50a0de590f98b6e3cd3f5ba4" exitCode=0
Oct 30 19:21:12.352800 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:12.352424    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82" event={"ID":"5ad27024-eaea-4355-8691-2ce6018b4c89","Type":"ContainerDied","Data":"4ad42dc321e0c19adf0392373efd3c5cefbe0dbd50a0de590f98b6e3cd3f5ba4"}
Oct 30 19:21:13.502406 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:13.502371    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:13.595434 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:13.595401    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad27024-eaea-4355-8691-2ce6018b4c89-kube-api-access-gmgxg\" (UniqueName: \"kubernetes.io/projected/5ad27024-eaea-4355-8691-2ce6018b4c89-kube-api-access-gmgxg\") pod \"5ad27024-eaea-4355-8691-2ce6018b4c89\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") "
Oct 30 19:21:13.595667 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:13.595479    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-openshift-ca\") pod \"5ad27024-eaea-4355-8691-2ce6018b4c89\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") "
Oct 30 19:21:13.595667 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:13.595545    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-xks-ca\") pod \"5ad27024-eaea-4355-8691-2ce6018b4c89\" (UID: \"5ad27024-eaea-4355-8691-2ce6018b4c89\") "
Oct 30 19:21:13.595862 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:13.595838    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-openshift-ca" pod "5ad27024-eaea-4355-8691-2ce6018b4c89" (UID: "5ad27024-eaea-4355-8691-2ce6018b4c89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:13.596039 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:13.596019    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-openshift-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:13.597650 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:13.597623    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-xks-ca" pod "5ad27024-eaea-4355-8691-2ce6018b4c89" (UID: "5ad27024-eaea-4355-8691-2ce6018b4c89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:13.597763 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:13.597685    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad27024-eaea-4355-8691-2ce6018b4c89-kube-api-access-gmgxg" pod "5ad27024-eaea-4355-8691-2ce6018b4c89" (UID: "5ad27024-eaea-4355-8691-2ce6018b4c89"). InnerVolumeSpecName "kube-api-access-gmgxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:13.696564 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:13.696529    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgxg\" (UniqueName: \"kubernetes.io/projected/5ad27024-eaea-4355-8691-2ce6018b4c89-kube-api-access-gmgxg\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:13.696564 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:13.696556    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad27024-eaea-4355-8691-2ce6018b4c89-maas-api-xks-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:14.366255 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:14.366222    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"
Oct 30 19:21:14.366432 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:14.366221    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82" event={"ID":"5ad27024-eaea-4355-8691-2ce6018b4c89","Type":"ContainerDied","Data":"e8b0b980c00ea14eb698ecd1610532652713eef48ae21ed67cce874d4c64773e"}
Oct 30 19:21:14.366432 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:14.366341    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8b0b980c00ea14eb698ecd1610532652713eef48ae21ed67cce874d4c64773e"
Oct 30 19:21:16.027768 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:16.027726    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x" podUID="c05e5c49-b96a-432f-8a57-669cde74121d" containerName="payload-pre-processing" containerID="cri-o://f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495" gracePeriod=25
Oct 30 19:21:25.299548 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:25.299509    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x" podUID="c05e5c49-b96a-432f-8a57-669cde74121d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:25.651036 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:25.650949    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:21:35.298853 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:35.298812    4391 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x" podUID="c05e5c49-b96a-432f-8a57-669cde74121d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:35.655055 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:35.655025    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"]
Oct 30 19:21:35.655264 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:35.655229    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerName="istio-proxy" containerID="cri-o://9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606" gracePeriod=30
Oct 30 19:21:36.280983 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.280955    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 19:21:36.343097 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.343026    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05e5c49-b96a-432f-8a57-669cde74121d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05e5c49-b96a-432f-8a57-669cde74121d-plugins-config-volume\") pod \"c05e5c49-b96a-432f-8a57-669cde74121d\" (UID: \"c05e5c49-b96a-432f-8a57-669cde74121d\") "
Oct 30 19:21:36.343467 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.343119    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05e5c49-b96a-432f-8a57-669cde74121d-kube-api-access-zdb7h\" (UniqueName: \"kubernetes.io/projected/c05e5c49-b96a-432f-8a57-669cde74121d-kube-api-access-zdb7h\") pod \"c05e5c49-b96a-432f-8a57-669cde74121d\" (UID: \"c05e5c49-b96a-432f-8a57-669cde74121d\") "
Oct 30 19:21:36.343467 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.343398    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05e5c49-b96a-432f-8a57-669cde74121d-plugins-config-volume" pod "c05e5c49-b96a-432f-8a57-669cde74121d" (UID: "c05e5c49-b96a-432f-8a57-669cde74121d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:36.345080 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.345060    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05e5c49-b96a-432f-8a57-669cde74121d-kube-api-access-zdb7h" pod "c05e5c49-b96a-432f-8a57-669cde74121d" (UID: "c05e5c49-b96a-432f-8a57-669cde74121d"). InnerVolumeSpecName "kube-api-access-zdb7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:36.444869 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.444833    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdb7h\" (UniqueName: \"kubernetes.io/projected/c05e5c49-b96a-432f-8a57-669cde74121d-kube-api-access-zdb7h\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:36.444869 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.444863    4391 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05e5c49-b96a-432f-8a57-669cde74121d-plugins-config-volume\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:36.487741 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.487707    4391 generic.go:356] "Generic (PLEG): container finished" podID="c05e5c49-b96a-432f-8a57-669cde74121d" containerID="f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495" exitCode=0
Oct 30 19:21:36.487918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.487760    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"
Oct 30 19:21:36.487918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.487797    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x" event={"ID":"c05e5c49-b96a-432f-8a57-669cde74121d","Type":"ContainerDied","Data":"f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495"}
Oct 30 19:21:36.487918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.487860    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x" event={"ID":"c05e5c49-b96a-432f-8a57-669cde74121d","Type":"ContainerDied","Data":"b4c4acb7c97c961bd0e1441343ca305c577cb59a2dadeb79bad9cb93b13dd14e"}
Oct 30 19:21:36.487918 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.487887    4391 scope.go:122] "RemoveContainer" containerID="f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495"
Oct 30 19:21:36.497364 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.497339    4391 scope.go:122] "RemoveContainer" containerID="f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495"
Oct 30 19:21:36.497645 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:21:36.497626    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495\": container with ID starting with f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495 not found: ID does not exist" containerID="f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495"
Oct 30 19:21:36.497710 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.497654    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495"} err="failed to get container status \"f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495\": rpc error: code = NotFound desc = could not find container \"f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495\": container with ID starting with f19368bfcb19e39aa6d5ccd5f8cdc6eb0bb188ad7a7ace306db77fda1a56d495 not found: ID does not exist"
Oct 30 19:21:36.647096 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.647000    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"]
Oct 30 19:21:36.649537 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:36.649510    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f5705f-569dfccdd7-l9w5x"]
Oct 30 19:21:37.005084 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:37.005052    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05e5c49-b96a-432f-8a57-669cde74121d" path="/var/lib/kubelet/pods/c05e5c49-b96a-432f-8a57-669cde74121d/volumes"
Oct 30 19:21:40.896211 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.896186    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 19:21:40.986802 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.986726    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istiod-ca-cert\") pod \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") "
Oct 30 19:21:40.986802 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.986763    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-podinfo\") pod \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") "
Oct 30 19:21:40.986802 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.986798    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-token\" (UniqueName: \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-token\") pod \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") "
Oct 30 19:21:40.987117 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.986831    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-socket\") pod \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") "
Oct 30 19:21:40.987117 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.986898    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-credential-socket\") pod \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") "
Oct 30 19:21:40.987117 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.986948    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-envoy\") pod \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") "
Oct 30 19:21:40.987117 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987012    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-data\") pod \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") "
Oct 30 19:21:40.987117 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987073    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-certs\") pod \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") "
Oct 30 19:21:40.987117 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987110    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-kube-api-access-wqj9n\" (UniqueName: \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-kube-api-access-wqj9n\") pod \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\" (UID: \"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a\") "
Oct 30 19:21:40.987444 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987156    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istiod-ca-cert" pod "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" (UID: "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:40.987444 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987286    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-socket" pod "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" (UID: "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:40.987444 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987300    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-data" pod "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" (UID: "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:40.987444 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987421    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-credential-socket" pod "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" (UID: "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:40.987785 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987452    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-certs" pod "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" (UID: "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:40.987785 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987575    4391 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istiod-ca-cert\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:40.987785 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987617    4391 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:40.987785 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987632    4391 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-credential-socket\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:40.987785 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987645    4391 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-data\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:40.987785 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.987659    4391 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-workload-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:40.989283 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.989260    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-kube-api-access-wqj9n" pod "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" (UID: "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a"). InnerVolumeSpecName "kube-api-access-wqj9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:40.989338 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.989270    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-token" pod "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" (UID: "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:40.989445 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.989426    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-envoy" pod "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" (UID: "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:40.989490 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:40.989451    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-podinfo" pod "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" (UID: "fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:41.088963 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.088930    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqj9n\" (UniqueName: \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-kube-api-access-wqj9n\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.088963 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.088958    4391 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-podinfo\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.088963 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.088968    4391 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-token\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.089191 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.088976    4391 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a-istio-envoy\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.254122 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:21:41.254024    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:21:41.524912 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.524816    4391 generic.go:356] "Generic (PLEG): container finished" podID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerID="9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606" exitCode=0
Oct 30 19:21:41.524912 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.524872    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"
Oct 30 19:21:41.525157 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.524913    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" event={"ID":"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a","Type":"ContainerDied","Data":"9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606"}
Oct 30 19:21:41.525157 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.524961    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj" event={"ID":"fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a","Type":"ContainerDied","Data":"9f654144afda55ce0c655464e97994c30e3015aea189ade5547d390f7ba67e1a"}
Oct 30 19:21:41.525157 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.524984    4391 scope.go:122] "RemoveContainer" containerID="9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606"
Oct 30 19:21:41.536998 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.536973    4391 scope.go:122] "RemoveContainer" containerID="9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606"
Oct 30 19:21:41.537315 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:21:41.537292    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606\": container with ID starting with 9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606 not found: ID does not exist" containerID="9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606"
Oct 30 19:21:41.537395 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.537328    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606"} err="failed to get container status \"9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606\": rpc error: code = NotFound desc = could not find container \"9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606\": container with ID starting with 9cd2f72d130a7a1d951fe9f2868da0a74f67fe665ef0b053e8986a2981281606 not found: ID does not exist"
Oct 30 19:21:41.683461 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.683429    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"]
Oct 30 19:21:41.730302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.730271    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-47wmj"]
Oct 30 19:21:41.751518 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:41.751480    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:21:43.009528 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:21:43.009489    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" path="/var/lib/kubelet/pods/fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a/volumes"
Oct 30 19:22:11.252765 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:22:11.252726    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:22:41.251826 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:22:41.251791    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:22:56.638575 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:22:56.638536    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:23:11.252480 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:23:11.252441    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:23:18.368218 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:23:18.368180    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:23:29.686138 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:23:29.686104    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:23:41.252230 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:23:41.252190    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:23:49.741144 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:23:49.741103    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:24:07.891184 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:24:07.891088    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:24:11.252543 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:24:11.252507    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:24:37.441551 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:24:37.441514    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:24:41.252942 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:24:41.252899    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:24:48.168822 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:24:48.168772    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:24:58.066836 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:24:58.066798    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:25:11.252084 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:25:11.252044    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:25:19.983604 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:25:19.983536    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:25:28.865363 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:25:28.865282    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:25:41.252266 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:25:41.252220    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:25:48.138089 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:25:48.138052    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:25:59.234620 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:25:59.234571    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:26:11.252767 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:26:11.252724    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:26:14.054885 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:26:14.054849    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"]
Oct 30 19:26:14.101215 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:26:14.101182    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be07cf67-brn82"]
Oct 30 19:26:15.006049 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:26:15.006004    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad27024-eaea-4355-8691-2ce6018b4c89" path="/var/lib/kubelet/pods/5ad27024-eaea-4355-8691-2ce6018b4c89/volumes"
Oct 30 19:26:19.973946 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:26:19.973894    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:26:39.037728 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:26:39.037695    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:26:41.255376 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:26:41.255321    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:26:51.933454 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:26:51.933413    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:27:01.559812 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:27:01.559732    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:27:11.252679 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:27:11.252558    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:27:22.488098 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:27:22.488064    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:27:31.954129 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:27:31.954092    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:27:41.252361 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:27:41.252324    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:27:46.472151 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:27:46.472118    4391 scope.go:122] "RemoveContainer" containerID="4ad42dc321e0c19adf0392373efd3c5cefbe0dbd50a0de590f98b6e3cd3f5ba4"
Oct 30 19:27:56.671728 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:27:56.671693    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:28:05.874349 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:28:05.874308    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:28:11.251797 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:28:11.251760    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:28:30.768007 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:28:30.767925    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:28:41.251787 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:28:41.251747    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:28:50.266348 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:28:50.266268    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:29:11.252562 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:29:11.252516    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:29:14.255272 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:29:14.255230    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:29:34.380176 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:29:34.380138    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:29:41.252548 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:29:41.252515    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:29:55.844486 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:29:55.844450    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:30:02.694253 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.694160    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"]
Oct 30 19:30:02.694766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.694640    4391 cpu_manager.go:414] "containerMap: removing container" podUID="c05e5c49-b96a-432f-8a57-669cde74121d" containerName="payload-pre-processing"
Oct 30 19:30:02.694766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.694654    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05e5c49-b96a-432f-8a57-669cde74121d" containerName="payload-pre-processing"
Oct 30 19:30:02.694766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.694666    4391 cpu_manager.go:414] "containerMap: removing container" podUID="5ad27024-eaea-4355-8691-2ce6018b4c89" containerName="revoke-keys"
Oct 30 19:30:02.694766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.694671    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad27024-eaea-4355-8691-2ce6018b4c89" containerName="revoke-keys"
Oct 30 19:30:02.694766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.694705    4391 cpu_manager.go:414] "containerMap: removing container" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerName="istio-proxy"
Oct 30 19:30:02.694766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.694710    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerName="istio-proxy"
Oct 30 19:30:02.694766 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.694769    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05e5c49-b96a-432f-8a57-669cde74121d" containerName="payload-pre-processing"
Oct 30 19:30:02.695003 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.694776    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad27024-eaea-4355-8691-2ce6018b4c89" containerName="revoke-keys"
Oct 30 19:30:02.695003 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.694790    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0c0eae-9f97-4972-894d-0ec6e6c2bb6a" containerName="istio-proxy"
Oct 30 19:30:02.698165 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.698139    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:02.747785 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.747749    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:30:02.794374 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.794340    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:30:02.796885 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.796860    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"]
Oct 30 19:30:02.841121 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.841089    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:30:02.841121 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.841089    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:30:02.849060 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.849032    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpcz4\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:02.849127 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.849087    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpcz4\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:02.849244 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.849230    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbmhp\" (UniqueName: \"kubernetes.io/projected/7b515ac1-d6f6-43fc-9915-37ef310107b0-kube-api-access-lbmhp\") pod \"maas-api-revoke-sub-3d4156f1-qpcz4\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:02.950412 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.950318    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpcz4\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:02.950617 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.950427    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbmhp\" (UniqueName: \"kubernetes.io/projected/7b515ac1-d6f6-43fc-9915-37ef310107b0-kube-api-access-lbmhp\") pod \"maas-api-revoke-sub-3d4156f1-qpcz4\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:02.950617 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.950470    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpcz4\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:02.951201 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.951178    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpcz4\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:02.952828 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:02.952801    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qpcz4\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:03.005021 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:03.004994    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbmhp\" (UniqueName: \"kubernetes.io/projected/7b515ac1-d6f6-43fc-9915-37ef310107b0-kube-api-access-lbmhp\") pod \"maas-api-revoke-sub-3d4156f1-qpcz4\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:03.010718 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:03.010689    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:03.376869 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:03.376811    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"]
Oct 30 19:30:03.379145 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:30:03.379118    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b515ac1_d6f6_43fc_9915_37ef310107b0.slice/crio-9f27a29b424572b0c4bff8736750c51cb30e24808103dbffd122ef7f9790cba7 WatchSource:0}: Error finding container 9f27a29b424572b0c4bff8736750c51cb30e24808103dbffd122ef7f9790cba7: Status 404 returned error can't find the container with id 9f27a29b424572b0c4bff8736750c51cb30e24808103dbffd122ef7f9790cba7
Oct 30 19:30:03.444197 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:03.444162    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4" event={"ID":"7b515ac1-d6f6-43fc-9915-37ef310107b0","Type":"ContainerStarted","Data":"9f27a29b424572b0c4bff8736750c51cb30e24808103dbffd122ef7f9790cba7"}
Oct 30 19:30:04.450863 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:04.450822    4391 generic.go:356] "Generic (PLEG): container finished" podID="7b515ac1-d6f6-43fc-9915-37ef310107b0" containerID="17465146b6d9cbf8c6328fe02b963860e961736e45518044097e2ee2a47c319c" exitCode=0
Oct 30 19:30:04.451306 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:04.450876    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4" event={"ID":"7b515ac1-d6f6-43fc-9915-37ef310107b0","Type":"ContainerDied","Data":"17465146b6d9cbf8c6328fe02b963860e961736e45518044097e2ee2a47c319c"}
Oct 30 19:30:05.597606 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:05.597568    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:05.690248 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:05.690210    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-openshift-ca\") pod \"7b515ac1-d6f6-43fc-9915-37ef310107b0\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") "
Oct 30 19:30:05.690424 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:05.690301    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b515ac1-d6f6-43fc-9915-37ef310107b0-kube-api-access-lbmhp\" (UniqueName: \"kubernetes.io/projected/7b515ac1-d6f6-43fc-9915-37ef310107b0-kube-api-access-lbmhp\") pod \"7b515ac1-d6f6-43fc-9915-37ef310107b0\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") "
Oct 30 19:30:05.690424 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:05.690416    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-xks-ca\") pod \"7b515ac1-d6f6-43fc-9915-37ef310107b0\" (UID: \"7b515ac1-d6f6-43fc-9915-37ef310107b0\") "
Oct 30 19:30:05.690678 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:05.690651    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-openshift-ca" pod "7b515ac1-d6f6-43fc-9915-37ef310107b0" (UID: "7b515ac1-d6f6-43fc-9915-37ef310107b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:05.690815 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:05.690790    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-openshift-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:05.692481 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:05.692453    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-xks-ca" pod "7b515ac1-d6f6-43fc-9915-37ef310107b0" (UID: "7b515ac1-d6f6-43fc-9915-37ef310107b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:05.692557 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:05.692497    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b515ac1-d6f6-43fc-9915-37ef310107b0-kube-api-access-lbmhp" pod "7b515ac1-d6f6-43fc-9915-37ef310107b0" (UID: "7b515ac1-d6f6-43fc-9915-37ef310107b0"). InnerVolumeSpecName "kube-api-access-lbmhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:05.791386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:05.791287    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b515ac1-d6f6-43fc-9915-37ef310107b0-maas-api-xks-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:05.791386 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:05.791324    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbmhp\" (UniqueName: \"kubernetes.io/projected/7b515ac1-d6f6-43fc-9915-37ef310107b0-kube-api-access-lbmhp\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:06.463431 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:06.463394    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4" event={"ID":"7b515ac1-d6f6-43fc-9915-37ef310107b0","Type":"ContainerDied","Data":"9f27a29b424572b0c4bff8736750c51cb30e24808103dbffd122ef7f9790cba7"}
Oct 30 19:30:06.463638 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:06.463448    4391 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f27a29b424572b0c4bff8736750c51cb30e24808103dbffd122ef7f9790cba7"
Oct 30 19:30:06.463638 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:06.463408    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"
Oct 30 19:30:11.252953 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:30:11.252911    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:30:18.671841 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:18.671806    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:30:28.567612 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:30:28.567563    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:30:41.253158 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:30:41.253115    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:31:11.252326 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:31:11.252282    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:31:41.251289 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:31:41.251254    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:32:01.147705 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:32:01.147669    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:32:11.252557 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:32:11.252513    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:32:18.491136 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:32:18.491096    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:32:41.257569 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:32:41.257523    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:32:57.858622 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:32:57.857667    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:33:11.253552 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:33:11.253510    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:33:16.196383 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:33:16.196345    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:33:40.840706 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:33:40.840665    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:33:41.252432 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:33:41.252396    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:33:55.182401 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:33:55.182364    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:34:11.253327 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:34:11.253291    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:34:13.382131 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:34:13.382092    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:34:28.857605 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:34:28.857496    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:34:41.251779 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:34:41.251738    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:34:52.136789 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:34:52.136755    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:35:06.046352 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:35:06.046312    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"]
Oct 30 19:35:06.092928 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:35:06.092896    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qpcz4"]
Oct 30 19:35:07.007274 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:35:07.007236    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b515ac1-d6f6-43fc-9915-37ef310107b0" path="/var/lib/kubelet/pods/7b515ac1-d6f6-43fc-9915-37ef310107b0/volumes"
Oct 30 19:35:11.252634 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:35:11.252562    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:35:16.980325 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:35:16.980286    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:35:32.275718 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:35:32.275685    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:35:41.251922 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:35:41.251885    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:35:42.484909 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:35:42.484874    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:35:54.580296 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:35:54.580259    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:36:03.463800 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:03.463762    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:36:11.254078 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:36:11.254035    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:36:25.139374 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:25.139339    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:36:26.062659 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.062620    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"]
Oct 30 19:36:26.063070 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.063058    4391 cpu_manager.go:414] "containerMap: removing container" podUID="7b515ac1-d6f6-43fc-9915-37ef310107b0" containerName="revoke-keys"
Oct 30 19:36:26.063123 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.063072    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b515ac1-d6f6-43fc-9915-37ef310107b0" containerName="revoke-keys"
Oct 30 19:36:26.063159 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.063152    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b515ac1-d6f6-43fc-9915-37ef310107b0" containerName="revoke-keys"
Oct 30 19:36:26.066658 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.066633    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.085874 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.085848    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7qt\" (UniqueName: \"kubernetes.io/projected/8523ba33-9669-4f23-b553-9ce9c39757cc-kube-api-access-lg7qt\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.086043 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.085969    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-metrics-certs\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.086043 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.086006    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-maas-api-tls\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.086172 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.086148    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8523ba33-9669-4f23-b553-9ce9c39757cc-openshift-service-ca\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.162864 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.162831    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:26.162864 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.162843    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:26.163320 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.163301    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 19:36:26.163465 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.163448    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:36:26.186824 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.186790    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8523ba33-9669-4f23-b553-9ce9c39757cc-openshift-service-ca\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.187013 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.186868    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7qt\" (UniqueName: \"kubernetes.io/projected/8523ba33-9669-4f23-b553-9ce9c39757cc-kube-api-access-lg7qt\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.187013 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.186952    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-metrics-certs\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.187013 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.186997    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-maas-api-tls\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.187595 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.187563    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8523ba33-9669-4f23-b553-9ce9c39757cc-openshift-service-ca\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.189421 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.189397    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-metrics-certs\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.189515 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.189431    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-maas-api-tls\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.209662 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.209639    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-977zh\""
Oct 30 19:36:26.212352 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.212331    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"]
Oct 30 19:36:26.258856 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.258824    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7qt\" (UniqueName: \"kubernetes.io/projected/8523ba33-9669-4f23-b553-9ce9c39757cc-kube-api-access-lg7qt\") pod \"maas-api-5c48877c75-hdfg4\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.379879 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.379802    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:26.756025 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.755996    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"]
Oct 30 19:36:26.757935 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:36:26.757888    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8523ba33_9669_4f23_b553_9ce9c39757cc.slice/crio-d1bc544adc3becd9aa73ce02d7d56beb3193a87f5afd1b83843af1e2a83e7f3d WatchSource:0}: Error finding container d1bc544adc3becd9aa73ce02d7d56beb3193a87f5afd1b83843af1e2a83e7f3d: Status 404 returned error can't find the container with id d1bc544adc3becd9aa73ce02d7d56beb3193a87f5afd1b83843af1e2a83e7f3d
Oct 30 19:36:26.758831 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:26.758812    4391 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:36:27.662322 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:27.662288    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4" event={"ID":"8523ba33-9669-4f23-b553-9ce9c39757cc","Type":"ContainerStarted","Data":"94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f"}
Oct 30 19:36:27.662725 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:27.662349    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4" event={"ID":"8523ba33-9669-4f23-b553-9ce9c39757cc","Type":"ContainerStarted","Data":"d1bc544adc3becd9aa73ce02d7d56beb3193a87f5afd1b83843af1e2a83e7f3d"}
Oct 30 19:36:27.662725 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:27.662492    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:27.797952 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:27.797890    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4" podStartSLOduration=0.977814591 podStartE2EDuration="1.79787431s" totalImagesPullingTime="820.059719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:27.796632646 +0000 UTC m=+3582.907365929" watchObservedRunningTime="2026-10-30 19:36:27.79787431 +0000 UTC m=+3582.908607593"
Oct 30 19:36:33.674003 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:33.673971    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:34.351048 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.351017    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"]
Oct 30 19:36:34.355026 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.355004    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.460063 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.460028    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"]
Oct 30 19:36:34.476034 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.475999    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d185fef-c86a-42f4-a83e-42e1b3da5806-metrics-certs\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.476210 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.476039    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d185fef-c86a-42f4-a83e-42e1b3da5806-openshift-service-ca\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.476210 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.476183    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d185fef-c86a-42f4-a83e-42e1b3da5806-maas-api-tls\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.476339 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.476237    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhn65\" (UniqueName: \"kubernetes.io/projected/2d185fef-c86a-42f4-a83e-42e1b3da5806-kube-api-access-qhn65\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.577001 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.576968    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d185fef-c86a-42f4-a83e-42e1b3da5806-maas-api-tls\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.577170 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.577037    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhn65\" (UniqueName: \"kubernetes.io/projected/2d185fef-c86a-42f4-a83e-42e1b3da5806-kube-api-access-qhn65\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.577170 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.577072    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d185fef-c86a-42f4-a83e-42e1b3da5806-metrics-certs\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.577170 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.577096    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d185fef-c86a-42f4-a83e-42e1b3da5806-openshift-service-ca\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.577828 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.577797    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d185fef-c86a-42f4-a83e-42e1b3da5806-openshift-service-ca\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.579520 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.579487    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d185fef-c86a-42f4-a83e-42e1b3da5806-maas-api-tls\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.579665 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.579530    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d185fef-c86a-42f4-a83e-42e1b3da5806-metrics-certs\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.632106 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.632034    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhn65\" (UniqueName: \"kubernetes.io/projected/2d185fef-c86a-42f4-a83e-42e1b3da5806-kube-api-access-qhn65\") pod \"maas-api-595cc648c-dzb2m\" (UID: \"2d185fef-c86a-42f4-a83e-42e1b3da5806\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:34.668540 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:34.668497    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:35.036191 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:35.036166    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"]
Oct 30 19:36:35.038248 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:36:35.038216    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d185fef_c86a_42f4_a83e_42e1b3da5806.slice/crio-ab67bf72e9477a704757714f216f453847a2050a2069e8c610f4823da2e449bc WatchSource:0}: Error finding container ab67bf72e9477a704757714f216f453847a2050a2069e8c610f4823da2e449bc: Status 404 returned error can't find the container with id ab67bf72e9477a704757714f216f453847a2050a2069e8c610f4823da2e449bc
Oct 30 19:36:35.705112 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:35.705077    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m" event={"ID":"2d185fef-c86a-42f4-a83e-42e1b3da5806","Type":"ContainerStarted","Data":"ab67bf72e9477a704757714f216f453847a2050a2069e8c610f4823da2e449bc"}
Oct 30 19:36:36.251833 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:36.251799    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:36:36.713598 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:36.713544    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m" event={"ID":"2d185fef-c86a-42f4-a83e-42e1b3da5806","Type":"ContainerStarted","Data":"f60bf992cd6d01af593f9fc2aefebf0f2daa2bce28f445b5933e633cea13a9da"}
Oct 30 19:36:36.713806 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:36.713711    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:41.251626 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:36:41.251570    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:36:42.728200 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:42.728173    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m"
Oct 30 19:36:42.837798 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:42.837746    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-dzb2m" podStartSLOduration=8.026060165 podStartE2EDuration="8.837734315s" totalImagesPullingTime="811.67415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:36.87301616 +0000 UTC m=+3591.983749444" watchObservedRunningTime="2026-10-30 19:36:42.837734315 +0000 UTC m=+3597.948467598"
Oct 30 19:36:42.886815 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:42.886779    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"]
Oct 30 19:36:42.887046 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:42.887024    4391 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4" podUID="8523ba33-9669-4f23-b553-9ce9c39757cc" containerName="maas-api" containerID="cri-o://94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f" gracePeriod=30
Oct 30 19:36:43.138466 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.138442    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:43.175663 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.175632    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-metrics-certs\") pod \"8523ba33-9669-4f23-b553-9ce9c39757cc\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") "
Oct 30 19:36:43.175799 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.175715    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8523ba33-9669-4f23-b553-9ce9c39757cc-kube-api-access-lg7qt\" (UniqueName: \"kubernetes.io/projected/8523ba33-9669-4f23-b553-9ce9c39757cc-kube-api-access-lg7qt\") pod \"8523ba33-9669-4f23-b553-9ce9c39757cc\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") "
Oct 30 19:36:43.175862 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.175807    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-maas-api-tls\") pod \"8523ba33-9669-4f23-b553-9ce9c39757cc\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") "
Oct 30 19:36:43.175919 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.175868    4391 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8523ba33-9669-4f23-b553-9ce9c39757cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8523ba33-9669-4f23-b553-9ce9c39757cc-openshift-service-ca\") pod \"8523ba33-9669-4f23-b553-9ce9c39757cc\" (UID: \"8523ba33-9669-4f23-b553-9ce9c39757cc\") "
Oct 30 19:36:43.176295 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.176257    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8523ba33-9669-4f23-b553-9ce9c39757cc-openshift-service-ca" pod "8523ba33-9669-4f23-b553-9ce9c39757cc" (UID: "8523ba33-9669-4f23-b553-9ce9c39757cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:43.177880 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.177851    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-maas-api-tls" pod "8523ba33-9669-4f23-b553-9ce9c39757cc" (UID: "8523ba33-9669-4f23-b553-9ce9c39757cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:43.178290 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.178264    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-metrics-certs" pod "8523ba33-9669-4f23-b553-9ce9c39757cc" (UID: "8523ba33-9669-4f23-b553-9ce9c39757cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:43.178383 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.178266    4391 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8523ba33-9669-4f23-b553-9ce9c39757cc-kube-api-access-lg7qt" pod "8523ba33-9669-4f23-b553-9ce9c39757cc" (UID: "8523ba33-9669-4f23-b553-9ce9c39757cc"). InnerVolumeSpecName "kube-api-access-lg7qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:43.277190 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.277101    4391 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg7qt\" (UniqueName: \"kubernetes.io/projected/8523ba33-9669-4f23-b553-9ce9c39757cc-kube-api-access-lg7qt\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:43.277190 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.277142    4391 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-maas-api-tls\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:43.277190 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.277160    4391 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8523ba33-9669-4f23-b553-9ce9c39757cc-openshift-service-ca\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:43.277190 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.277173    4391 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8523ba33-9669-4f23-b553-9ce9c39757cc-metrics-certs\") on node \"ip-10-0-128-92.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:43.762029 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.761993    4391 generic.go:356] "Generic (PLEG): container finished" podID="8523ba33-9669-4f23-b553-9ce9c39757cc" containerID="94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f" exitCode=0
Oct 30 19:36:43.762466 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.762042    4391 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"
Oct 30 19:36:43.762466 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.762072    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4" event={"ID":"8523ba33-9669-4f23-b553-9ce9c39757cc","Type":"ContainerDied","Data":"94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f"}
Oct 30 19:36:43.762466 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.762124    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4" event={"ID":"8523ba33-9669-4f23-b553-9ce9c39757cc","Type":"ContainerDied","Data":"d1bc544adc3becd9aa73ce02d7d56beb3193a87f5afd1b83843af1e2a83e7f3d"}
Oct 30 19:36:43.762466 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.762136    4391 scope.go:122] "RemoveContainer" containerID="94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f"
Oct 30 19:36:43.772276 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.772258    4391 scope.go:122] "RemoveContainer" containerID="94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f"
Oct 30 19:36:43.772556 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:36:43.772536    4391 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f\": container with ID starting with 94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f not found: ID does not exist" containerID="94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f"
Oct 30 19:36:43.772627 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.772565    4391 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f"} err="failed to get container status \"94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f\": rpc error: code = NotFound desc = could not find container \"94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f\": container with ID starting with 94c067cdbfddbf5ff6384fddd46759d059e20a1bdd1862a860935368fd4d294f not found: ID does not exist"
Oct 30 19:36:43.919098 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.919053    4391 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"]
Oct 30 19:36:43.965717 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:43.965678    4391 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-hdfg4"]
Oct 30 19:36:45.005155 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:45.005122    4391 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8523ba33-9669-4f23-b553-9ce9c39757cc" path="/var/lib/kubelet/pods/8523ba33-9669-4f23-b553-9ce9c39757cc/volumes"
Oct 30 19:36:46.561641 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:46.561608    4391 scope.go:122] "RemoveContainer" containerID="17465146b6d9cbf8c6328fe02b963860e961736e45518044097e2ee2a47c319c"
Oct 30 19:36:47.391797 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:36:47.391760    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:37:08.543638 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:08.543601    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:37:11.252133 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:37:11.252091    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:37:40.463608 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:40.463546    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wrx5m"]
Oct 30 19:37:41.321135 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:37:41.320795    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:37:52.395212 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:52.395176    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-gmjrz_8ad60ab5-d7d1-4857-9df0-645b4bdc607d/manager/0.log"
Oct 30 19:37:52.695352 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:52.695321    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-cjtr8_9bbf302c-76be-47e8-84fd-1cc70e3f99df/manager/0.log"
Oct 30 19:37:53.024544 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:53.024459    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rv5c2_237bc5bb-3f41-4925-8344-939672e5ed5a/manager/0.log"
Oct 30 19:37:54.013658 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:54.013626    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jtmwb_8a6fb2a0-b06b-4a31-b663-2e7ec491a21a/server/0.log"
Oct 30 19:37:54.251602 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:54.251555    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-4dltq_cd502785-059e-4fc7-acf0-af91400f840e/manager/0.log"
Oct 30 19:37:54.584894 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:54.584866    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-r55t4_34aaf976-fadc-4922-878c-626c5841487b/manager/0.log"
Oct 30 19:37:55.542045 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:55.542014    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s_66e98c62-ea4d-4305-b143-61c2ce132dcc/manager/0.log"
Oct 30 19:37:55.612762 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:55.612732    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s_66e98c62-ea4d-4305-b143-61c2ce132dcc/copy-manifests/0.log"
Oct 30 19:37:57.043550 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:37:57.043519    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-dzb2m_2d185fef-c86a-42f4-a83e-42e1b3da5806/maas-api/0.log"
Oct 30 19:38:01.489788 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:01.489756    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-f4lk7_1ceaadbb-b694-4ddb-8847-6cd259db3a5e/manager/0.log"
Oct 30 19:38:02.448635 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:02.448575    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wrx5m_6e8e6505-6761-4a7a-bd39-b019b66d5db0/limitador/0.log"
Oct 30 19:38:03.508397 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:03.508361    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj_59262255-6bf4-49dc-86c2-db46d56acc3f/istio-proxy/0.log"
Oct 30 19:38:03.801217 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:03.801135    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-27pn2_a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25/discovery/0.log"
Oct 30 19:38:04.125055 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:04.124984    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58d577dc79-jtwnc_0196e473-3f42-4506-bedd-da006253b9de/kube-auth-proxy/0.log"
Oct 30 19:38:04.840271 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:04.840229    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-59j4p_dc4a0ecf-d63c-4917-95e5-44998c76ef0a/istio-proxy/0.log"
Oct 30 19:38:05.622260 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:05.622228    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-gpdv4_ef6090f1-2038-4579-9627-a1309e77367b/istio-proxy/0.log"
Oct 30 19:38:06.650516 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:06.650485    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t8j9r_d4909aee-0ac5-4527-a9d9-976006e9a1cc/istio-proxy/0.log"
Oct 30 19:38:11.252134 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:38:11.252092    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:38:21.421681 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:21.421652    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-gmjrz_8ad60ab5-d7d1-4857-9df0-645b4bdc607d/manager/0.log"
Oct 30 19:38:21.717254 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:21.717183    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-cjtr8_9bbf302c-76be-47e8-84fd-1cc70e3f99df/manager/0.log"
Oct 30 19:38:22.046984 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:22.046903    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rv5c2_237bc5bb-3f41-4925-8344-939672e5ed5a/manager/0.log"
Oct 30 19:38:22.746353 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:22.746324    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jtmwb_8a6fb2a0-b06b-4a31-b663-2e7ec491a21a/server/0.log"
Oct 30 19:38:22.986448 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:22.986414    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-4dltq_cd502785-059e-4fc7-acf0-af91400f840e/manager/0.log"
Oct 30 19:38:23.285568 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:23.285534    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-r55t4_34aaf976-fadc-4922-878c-626c5841487b/manager/0.log"
Oct 30 19:38:24.314948 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:24.314917    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s_66e98c62-ea4d-4305-b143-61c2ce132dcc/copy-manifests/0.log"
Oct 30 19:38:24.384529 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:24.384498    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9c7b5ddd8-v8f8s_66e98c62-ea4d-4305-b143-61c2ce132dcc/manager/0.log"
Oct 30 19:38:25.534425 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:25.534395    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-dzb2m_2d185fef-c86a-42f4-a83e-42e1b3da5806/maas-api/0.log"
Oct 30 19:38:29.582922 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:29.582888    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-f4lk7_1ceaadbb-b694-4ddb-8847-6cd259db3a5e/manager/0.log"
Oct 30 19:38:30.480723 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:30.480687    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wrx5m_6e8e6505-6761-4a7a-bd39-b019b66d5db0/limitador/0.log"
Oct 30 19:38:31.375807 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:31.375773    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj_59262255-6bf4-49dc-86c2-db46d56acc3f/istio-proxy/0.log"
Oct 30 19:38:31.667087 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:31.667057    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-27pn2_a2a814f8-0aeb-4ba5-895e-8fbf7ed57d25/discovery/0.log"
Oct 30 19:38:31.991207 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:31.991130    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58d577dc79-jtwnc_0196e473-3f42-4506-bedd-da006253b9de/kube-auth-proxy/0.log"
Oct 30 19:38:32.684564 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:32.684534    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-59j4p_dc4a0ecf-d63c-4917-95e5-44998c76ef0a/istio-proxy/0.log"
Oct 30 19:38:33.414476 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:33.414449    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-gpdv4_ef6090f1-2038-4579-9627-a1309e77367b/istio-proxy/0.log"
Oct 30 19:38:34.449837 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:38:34.449807    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t8j9r_d4909aee-0ac5-4527-a9d9-976006e9a1cc/istio-proxy/0.log"
Oct 30 19:38:41.252857 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:38:41.252817    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:39:11.252506 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:39:11.252422    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:39:41.252297 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:39:41.252259    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:40:01.886251 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:01.886217    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-55w4h_2601c6f1-0eb6-4038-b54c-5f8a7214db42/global-pull-secret-syncer/0.log"
Oct 30 19:40:06.117769 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:06.117733    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rqbk8_a57df4ae-cbc2-41ac-ab43-c7c21cbe65d9/konnectivity-agent/0.log"
Oct 30 19:40:06.539668 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:06.539633    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-92.us-west-2.compute.internal_4ce1444a469ee2b4c13e78a90a9b0681/haproxy/0.log"
Oct 30 19:40:10.844174 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:10.844142    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-f4lk7_1ceaadbb-b694-4ddb-8847-6cd259db3a5e/manager/0.log"
Oct 30 19:40:11.252500 ip-10-0-128-92 kubenswrapper[4391]: E1030 19:40:11.252465    4391 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2454l" containerName="extractor"
Oct 30 19:40:11.317838 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.317804    4391 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"]
Oct 30 19:40:11.318260 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.318247    4391 cpu_manager.go:414] "containerMap: removing container" podUID="8523ba33-9669-4f23-b553-9ce9c39757cc" containerName="maas-api"
Oct 30 19:40:11.318311 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.318262    4391 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8523ba33-9669-4f23-b553-9ce9c39757cc" containerName="maas-api"
Oct 30 19:40:11.318347 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.318343    4391 memory_manager.go:365] "RemoveStaleState removing state" podUID="8523ba33-9669-4f23-b553-9ce9c39757cc" containerName="maas-api"
Oct 30 19:40:11.321426 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.321403    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.375432 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.375405    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"]
Oct 30 19:40:11.397741 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.397715    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-lib-modules\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.397880 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.397768    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-sys\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.397928 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.397873    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-proc\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.397966 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.397931    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj8pk\" (UniqueName: \"kubernetes.io/projected/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-kube-api-access-fj8pk\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.398001 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.397982    4391 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-podres\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.419028 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.419003    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cx52t\"/\"openshift-service-ca.crt\""
Oct 30 19:40:11.419153 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.419094    4391 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cx52t\"/\"kube-root-ca.crt\""
Oct 30 19:40:11.419310 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.419297    4391 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cx52t\"/\"default-dockercfg-76gpr\""
Oct 30 19:40:11.498802 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.498770    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-lib-modules\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.498973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.498829    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-sys\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.498973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.498881    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-proc\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.498973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.498911    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-lib-modules\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.498973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.498930    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj8pk\" (UniqueName: \"kubernetes.io/projected/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-kube-api-access-fj8pk\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.498973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.498954    4391 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-podres\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.498973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.498959    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-sys\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.498973 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.498971    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-proc\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.499218 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.499030    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-podres\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.552679 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.552608    4391 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj8pk\" (UniqueName: \"kubernetes.io/projected/f0e85c22-95c3-4f02-b5d0-fc2bb086eba1-kube-api-access-fj8pk\") pod \"perf-node-gather-daemonset-2pqlc\" (UID: \"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:11.633788 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:11.633755    4391 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:12.028626 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:12.028594    4391 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"]
Oct 30 19:40:12.031273 ip-10-0-128-92 kubenswrapper[4391]: W1030 19:40:12.031243    4391 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf0e85c22_95c3_4f02_b5d0_fc2bb086eba1.slice/crio-e0ae8ce326ae39837377c652cd7fa62809f7f689a404f1802b8e7feba25ed5f8 WatchSource:0}: Error finding container e0ae8ce326ae39837377c652cd7fa62809f7f689a404f1802b8e7feba25ed5f8: Status 404 returned error can't find the container with id e0ae8ce326ae39837377c652cd7fa62809f7f689a404f1802b8e7feba25ed5f8
Oct 30 19:40:12.238281 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:12.238248    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wrx5m_6e8e6505-6761-4a7a-bd39-b019b66d5db0/limitador/0.log"
Oct 30 19:40:12.945852 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:12.945819    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc" event={"ID":"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1","Type":"ContainerStarted","Data":"7f438ceace296d2ea46d6b3c077e654d7ae4a6c0febdf1afd57b756cc1a3e8c9"}
Oct 30 19:40:12.946052 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:12.945877    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:12.946052 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:12.945897    4391 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc" event={"ID":"f0e85c22-95c3-4f02-b5d0-fc2bb086eba1","Type":"ContainerStarted","Data":"e0ae8ce326ae39837377c652cd7fa62809f7f689a404f1802b8e7feba25ed5f8"}
Oct 30 19:40:13.056315 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:13.056258    4391 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc" podStartSLOduration=2.05624238 podStartE2EDuration="2.05624238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:13.055666078 +0000 UTC m=+3808.166399360" watchObservedRunningTime="2026-10-30 19:40:13.05624238 +0000 UTC m=+3808.166975663"
Oct 30 19:40:15.202217 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:15.202157    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4e1f4f57-65ab-421c-a873-ce48186c5115/alertmanager/0.log"
Oct 30 19:40:15.612948 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:15.612869    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4e1f4f57-65ab-421c-a873-ce48186c5115/config-reloader/0.log"
Oct 30 19:40:16.021060 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:16.021031    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4e1f4f57-65ab-421c-a873-ce48186c5115/kube-rbac-proxy-web/0.log"
Oct 30 19:40:16.429801 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:16.429776    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4e1f4f57-65ab-421c-a873-ce48186c5115/kube-rbac-proxy/0.log"
Oct 30 19:40:16.836140 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:16.836068    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4e1f4f57-65ab-421c-a873-ce48186c5115/kube-rbac-proxy-metric/0.log"
Oct 30 19:40:17.261777 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:17.261748    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4e1f4f57-65ab-421c-a873-ce48186c5115/prom-label-proxy/0.log"
Oct 30 19:40:17.673642 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:17.673614    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4e1f4f57-65ab-421c-a873-ce48186c5115/init-config-reloader/0.log"
Oct 30 19:40:18.509109 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:18.509073    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fpmkc_0ee5578e-70b9-45ad-b720-7d8bd3a2bda9/kube-state-metrics/0.log"
Oct 30 19:40:18.919387 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:18.919360    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fpmkc_0ee5578e-70b9-45ad-b720-7d8bd3a2bda9/kube-rbac-proxy-main/0.log"
Oct 30 19:40:18.961514 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:18.961490    4391 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-2pqlc"
Oct 30 19:40:19.331548 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:19.331471    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fpmkc_0ee5578e-70b9-45ad-b720-7d8bd3a2bda9/kube-rbac-proxy-self/0.log"
Oct 30 19:40:19.742185 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:19.742147    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-56d957c68c-qcj58_2ca3276a-5fe8-4978-97b2-79098bb956d3/metrics-server/0.log"
Oct 30 19:40:20.158116 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:20.158086    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-hz5k9_cbb0e89e-34f4-4ee5-adca-8f086abc024c/monitoring-plugin/0.log"
Oct 30 19:40:20.573436 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:20.573355    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cmz4n_7e2c5a8a-5c17-497f-a042-268d9481f8f1/node-exporter/0.log"
Oct 30 19:40:20.981447 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:20.981398    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cmz4n_7e2c5a8a-5c17-497f-a042-268d9481f8f1/kube-rbac-proxy/0.log"
Oct 30 19:40:21.393761 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:21.393737    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cmz4n_7e2c5a8a-5c17-497f-a042-268d9481f8f1/init-textfile/0.log"
Oct 30 19:40:24.302746 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:24.302695    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jmrdq_80d7ed04-4512-4e3e-80e4-b880915f0d9b/kube-rbac-proxy-main/0.log"
Oct 30 19:40:24.711110 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:24.711082    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jmrdq_80d7ed04-4512-4e3e-80e4-b880915f0d9b/kube-rbac-proxy-self/0.log"
Oct 30 19:40:25.118048 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:25.117963    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jmrdq_80d7ed04-4512-4e3e-80e4-b880915f0d9b/openshift-state-metrics/0.log"
Oct 30 19:40:29.269936 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:29.269848    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-9999x_09ed0cc6-ba84-46cf-8b98-f2880f2d987b/prometheus-operator-admission-webhook/0.log"
Oct 30 19:40:29.684936 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:29.684905    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cf66459f-jhs5g_195b50cc-7fbc-451e-b2c7-cea82d04d2f8/telemeter-client/0.log"
Oct 30 19:40:30.094542 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:30.094462    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cf66459f-jhs5g_195b50cc-7fbc-451e-b2c7-cea82d04d2f8/reload/0.log"
Oct 30 19:40:30.503302 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:30.503264    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cf66459f-jhs5g_195b50cc-7fbc-451e-b2c7-cea82d04d2f8/kube-rbac-proxy/0.log"
Oct 30 19:40:30.917781 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:30.917756    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c7bb6cbd-wnchg_3f08cc73-2de9-403d-bf75-afecaf1d8b63/thanos-query/0.log"
Oct 30 19:40:31.327924 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:31.327846    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c7bb6cbd-wnchg_3f08cc73-2de9-403d-bf75-afecaf1d8b63/kube-rbac-proxy-web/0.log"
Oct 30 19:40:31.736765 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:31.736736    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c7bb6cbd-wnchg_3f08cc73-2de9-403d-bf75-afecaf1d8b63/kube-rbac-proxy/0.log"
Oct 30 19:40:32.144556 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:32.144485    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c7bb6cbd-wnchg_3f08cc73-2de9-403d-bf75-afecaf1d8b63/prom-label-proxy/0.log"
Oct 30 19:40:32.562034 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:32.562009    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c7bb6cbd-wnchg_3f08cc73-2de9-403d-bf75-afecaf1d8b63/kube-rbac-proxy-rules/0.log"
Oct 30 19:40:32.975913 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:32.975880    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c7bb6cbd-wnchg_3f08cc73-2de9-403d-bf75-afecaf1d8b63/kube-rbac-proxy-metrics/0.log"
Oct 30 19:40:33.948777 ip-10-0-128-92 kubenswrapper[4391]: I1030 19:40:33.948747    4391 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7cdkfbj_59262255-6bf4-49dc-86c2-db46d56acc3f/istio-proxy/0.log"