Sep 18 13:44:16.365796 ip-10-0-143-135 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 13:44:16.710641 ip-10-0-143-135 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:44:16.710641 ip-10-0-143-135 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 13:44:16.710641 ip-10-0-143-135 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:44:16.710641 ip-10-0-143-135 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 13:44:16.710641 ip-10-0-143-135 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:44:16.711531 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.711436    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 13:44:16.714418 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714403    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714420    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714424    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714427    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714430    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714433    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714435    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714438    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714440    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714443    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714445    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714448    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714450    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714453    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714457    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714460    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:16.714453 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714463    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714465    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714472    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714475    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714478    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714480    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714483    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714485    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714487    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714490    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714492    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714495    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714511    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714513    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714516    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714518    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714521    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714523    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714526    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714528    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:16.714839 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714530    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714533    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714535    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714537    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714540    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714542    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714545    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714549    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714552    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714555    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714557    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714560    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714562    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714564    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714567    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714570    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714572    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714575    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714578    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714580    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:16.715302 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714582    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714585    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714587    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714589    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714592    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714594    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714597    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714599    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714601    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714604    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714607    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714609    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714611    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714614    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714616    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714619    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714621    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714624    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714626    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714628    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:16.715848 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714632    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714636    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714638    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714642    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714645    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714647    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714651    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714653    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714656    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714658    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714661    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714663    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714666    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714669    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714672    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714674    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714677    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714680    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714683    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:16.716305 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714686    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714689    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714691    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.714694    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715324    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715329    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715332    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715335    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715337    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715346    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715349    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715352    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715355    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715357    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715360    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715362    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715365    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715367    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715370    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715373    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:16.716764 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715376    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715378    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715382    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715386    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715388    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715391    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715394    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715396    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715399    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715402    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715404    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715407    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715409    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715412    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715414    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715417    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715419    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715422    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715424    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:16.717315 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715426    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715430    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715433    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715437    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715440    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715443    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715446    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715449    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715453    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715456    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715458    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715461    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715465    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715468    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715470    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715473    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715476    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715479    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715481    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:16.717787 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715484    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715486    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715489    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715492    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715495    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715513    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715516    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715519    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715521    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715524    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715527    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715529    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715532    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715534    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715537    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715539    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715541    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715544    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715547    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715549    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:16.718221 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715552    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715554    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715557    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715559    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715561    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715564    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715568    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715570    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715573    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715575    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715578    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715580    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715583    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715585    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715595    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715598    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715600    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715603    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715605    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715607    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:16.718711 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715610    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715612    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715614    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715617    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715619    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715966    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715971    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715974    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715976    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715979    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715982    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715985    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715987    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715990    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715992    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715996    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.715999    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716003    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716005    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716008    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:16.719184 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716011    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716013    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716016    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716018    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716020    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716023    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716025    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716028    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716031    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716033    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716036    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716038    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716040    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716043    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716045    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716047    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716050    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716052    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716055    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716057    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:16.719682 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716059    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716062    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716071    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716074    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716077    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716079    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716081    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716087    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716089    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716092    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716094    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716096    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716099    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716102    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716104    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716107    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716109    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716111    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716114    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716117    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:16.720142 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716119    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716121    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716124    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716126    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716128    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716131    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716133    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716135    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716138    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716140    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716142    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716145    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716147    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716151    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716154    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716157    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716159    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716161    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716164    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716167    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:16.720651 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716170    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716172    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716175    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716179    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716182    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716185    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716188    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716190    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716193    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716196    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716198    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716201    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716203    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716205    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716208    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716210    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716212    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716215    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716217    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:16.721115 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716220    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716223    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716226    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716228    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716230    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716294    2569 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716326    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716334    2569 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716339    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716353    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716358    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716366    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716375    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716378    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716381    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716384    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716387    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716390    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716393    2569 flags.go:64] FLAG: --cgroup-root=""
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716396    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716398    2569 flags.go:64] FLAG: --client-ca-file=""
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716401    2569 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716404    2569 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 13:44:16.721558 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716407    2569 flags.go:64] FLAG: --cluster-domain=""
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716410    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716413    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716416    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716419    2569 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716422    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716425    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716428    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716431    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716434    2569 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716436    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716439    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716442    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716445    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716448    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716452    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716455    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716458    2569 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716460    2569 flags.go:64] FLAG: --enable-server="true"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716469    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716475    2569 flags.go:64] FLAG: --event-burst="100"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716479    2569 flags.go:64] FLAG: --event-qps="50"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716482    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716485    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 13:44:16.722089 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716488    2569 flags.go:64] FLAG: --eviction-hard=""
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716491    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716494    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716512    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716515    2569 flags.go:64] FLAG: --eviction-soft=""
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716518    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716521    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716524    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716526    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716529    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716532    2569 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716535    2569 flags.go:64] FLAG: --feature-gates=""
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716538    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716546    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716549    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716552    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716555    2569 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716558    2569 flags.go:64] FLAG: --help="false"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716561    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-143-135.ec2.internal"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716564    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716567    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716570    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716573    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 13:44:16.722648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716578    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716581    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716583    2569 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716586    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716589    2569 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716594    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716598    2569 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716601    2569 flags.go:64] FLAG: --kube-reserved=""
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716604    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716606    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716609    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716612    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716615    2569 flags.go:64] FLAG: --lock-file=""
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716617    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716620    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716623    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716628    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716631    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716633    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716636    2569 flags.go:64] FLAG: --logging-format="text"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716639    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716642    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716645    2569 flags.go:64] FLAG: --manifest-url=""
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716647    2569 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716651    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 13:44:16.723609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716654    2569 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716657    2569 flags.go:64] FLAG: --max-pods="110"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716660    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716663    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716665    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716668    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716671    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716675    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716678    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716684    2569 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716687    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716690    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716693    2569 flags.go:64] FLAG: --pod-cidr=""
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716697    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716702    2569 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716704    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716707    2569 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716710    2569 flags.go:64] FLAG: --port="10250"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716713    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716716    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c87e8188b7eb5e5b"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716718    2569 flags.go:64] FLAG: --qos-reserved=""
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716721    2569 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716724    2569 flags.go:64] FLAG: --register-node="true"
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716727    2569 flags.go:64] FLAG: --register-with-taints=""
Sep 18 13:44:16.724679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716730    2569 flags.go:64] FLAG: --registry-burst="10"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716733    2569 flags.go:64] FLAG: --registry-qps="5"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716735    2569 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716738    2569 flags.go:64] FLAG: --reserved-memory=""
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716741    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716744    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716747    2569 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716750    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716752    2569 flags.go:64] FLAG: --runonce="false"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716755    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716758    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716760    2569 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716763    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716766    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716769    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716773    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716776    2569 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716779    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716782    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716784    2569 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716787    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716790    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716794    2569 flags.go:64] FLAG: --system-cgroups=""
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716797    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716802    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 13:44:16.725352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716805    2569 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716807    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716813    2569 flags.go:64] FLAG: --tls-min-version=""
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716816    2569 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716819    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716821    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716824    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716826    2569 flags.go:64] FLAG: --v="2"
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716830    2569 flags.go:64] FLAG: --version="false"
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716834    2569 flags.go:64] FLAG: --vmodule=""
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716837    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.716840    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716958    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716962    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716964    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716967    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716970    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716972    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716974    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716977    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716979    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716981    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716984    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:16.726249 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716987    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716990    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716992    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716995    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716997    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.716999    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717003    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717006    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717008    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717010    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717013    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717015    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717017    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717020    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717022    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717024    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717027    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717029    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717032    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717034    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:16.727160 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717036    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717039    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717043    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717047    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717050    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717052    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717055    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717058    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717060    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717063    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717066    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717068    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717071    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717073    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717076    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717078    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717080    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717082    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717086    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717088    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:16.727706 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717091    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717094    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717096    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717099    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717101    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717103    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717106    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717108    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717111    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717113    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717116    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717118    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717120    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717123    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717125    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717127    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717130    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717132    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717136    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:16.728530 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717139    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717142    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717144    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717147    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717150    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717152    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717155    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717157    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717159    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717162    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717164    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717168    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717171    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717174    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717178    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717181    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717183    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717185    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717188    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717191    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:16.729182 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717193    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717195    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717198    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717200    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717202    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717205    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717207    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717210    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.717212    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.717218    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.724860    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.724878    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.725632    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.725647    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725739    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:16.729766 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725747    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725752    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725757    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725761    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725765    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725769    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725773    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725777    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725784    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725790    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725795    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725800    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725804    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725808    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725813    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725817    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725822    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725826    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725830    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725834    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:16.730133 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725838    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725842    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725846    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725850    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725855    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725859    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725865    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725871    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725876    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725881    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725885    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725889    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725893    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725897    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725901    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725905    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725908    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725913    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725916    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:16.730633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725920    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725924    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725928    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725932    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725936    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725940    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725944    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725949    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725953    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725957    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725961    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725965    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725969    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725973    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725977    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725981    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725985    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725990    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725994    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.725998    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:16.731086 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726002    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726006    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726010    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726014    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726018    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726022    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726026    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726029    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726034    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726038    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726042    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726046    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726050    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726054    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726058    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726063    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726067    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726071    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726075    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726079    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:16.731577 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726083    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726087    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726091    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726095    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726100    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726104    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726109    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726116    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726120    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726124    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726128    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726132    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726135    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726140    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726144    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726148    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726153    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726157    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:16.732045 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726161    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.726168    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726333    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726341    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726345    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726349    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726354    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726358    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726362    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726366    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726370    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726375    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726379    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726383    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726387    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:44:16.732467 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726391    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726395    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726398    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726402    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726406    2569 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726410    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726414    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726418    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726423    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726427    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726431    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726435    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726438    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726442    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726446    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726451    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726455    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726459    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726462    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726467    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:44:16.732926 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726472    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726475    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726479    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726482    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726487    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726491    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726495    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726519    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726523    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726527    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726531    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726535    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726539    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726543    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726547    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726550    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726554    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726558    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726562    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726566    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:44:16.733697 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726570    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726574    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726578    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726581    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726586    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726590    2569 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726593    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726597    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726602    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726606    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726610    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726614    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726618    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726622    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726626    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726632    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726638    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726643    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726648    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:44:16.734347 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726652    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726656    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726660    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726664    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726667    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726671    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726675    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726679    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726683    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726687    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726690    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726694    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726698    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726702    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726706    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726710    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726714    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726718    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726722    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726726    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:44:16.734915 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726730    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726734    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726738    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726742    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726746    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726750    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:16.726755    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.726763    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.726902    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.729714    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.730598    2569 server.go:1019] "Starting client certificate rotation"
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.730697    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.730736    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:44:16.735457 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.734227    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:44:16.735844 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.735333    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:44:16.745421 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.745404    2569 log.go:25] "Validated CRI v1 runtime API"
Sep 18 13:44:16.747129 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.747114    2569 log.go:25] "Validated CRI v1 image API"
Sep 18 13:44:16.747915 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.747901    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 13:44:16.749988 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.749968    2569 fs.go:135] Filesystem UUIDs: map[77f0daa2-0b4d-4aff-b151-cbbe8eaac57a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7a7a9627-4925-42ca-b247-04fb21f8e2ec:/dev/nvme0n1p4]
Sep 18 13:44:16.750071 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.749987    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 13:44:16.755600 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.755466    2569 manager.go:216] Machine: {Timestamp:2026-09-18 13:44:16.754101291 +0000 UTC m=+0.284126849 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3197500 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29241e1b12412db9b3fd00783b4f05 SystemUUID:ec29241e-1b12-412d-b9b3-fd00783b4f05 BootID:868cca3d-4772-41fc-b233-a01ef3203841 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:30:5d:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:30:5d:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:ec:b5:64:85:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 13:44:16.755600 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.755594    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 13:44:16.755758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.755693    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 13:44:16.758488 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.758468    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 13:44:16.758631 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.758492    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-135.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 13:44:16.759230 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.759218    2569 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 13:44:16.759270 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.759235    2569 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 13:44:16.759270 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.759243    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:44:16.759270 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.759258    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:44:16.759346 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.759330    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 13:44:16.763037 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.763024    2569 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:44:16.763158 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.763148    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 13:44:16.763202 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.763178    2569 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 13:44:16.763782 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.763767    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 13:44:16.763836 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.763798    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 13:44:16.763836 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.763810    2569 kubelet.go:398] "Adding apiserver pod source"
Sep 18 13:44:16.763836 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.763818    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 13:44:16.764216 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.764203    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:44:16.764255 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.764228    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:44:16.764410 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.764394    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:44:16.766294 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.766271    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 13:44:16.771119 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.771084    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 13:44:16.771212 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.771140    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 13:44:16.772287 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772272    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 13:44:16.772352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772293    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 13:44:16.772352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772301    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 13:44:16.772352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772309    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 13:44:16.772352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772322    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 13:44:16.772352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772330    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 13:44:16.772352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772349    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 13:44:16.772571 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772357    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 13:44:16.772571 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772368    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 13:44:16.772571 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772377    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 13:44:16.772571 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772389    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 13:44:16.772571 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.772397    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 13:44:16.773157 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.773146    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 13:44:16.773201 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.773160    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 13:44:16.774485 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.774459    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-135.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 13:44:16.775225 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.775203    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 13:44:16.775269 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.775205    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-135.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 13:44:16.776900 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.776884    2569 server.go:1268] "Started kubelet"
Sep 18 13:44:16.777003 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.776926    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 13:44:16.777003 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.776994    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 13:44:16.777082 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.777029    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 13:44:16.777639 ip-10-0-143-135 systemd[1]: Started Kubernetes Kubelet.
Sep 18 13:44:16.778544 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.778318    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 13:44:16.779180 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.779167    2569 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 13:44:16.780731 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.780713    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 13:44:16.781913 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.781144    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-135.ec2.internal.18d66e308fd1682b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-135.ec2.internal,UID:ip-10-0-143-135.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-135.ec2.internal,},FirstTimestamp:2026-09-18 13:44:16.776701995 +0000 UTC m=+0.306727554,LastTimestamp:2026-09-18 13:44:16.776701995 +0000 UTC m=+0.306727554,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-135.ec2.internal,}"
Sep 18 13:44:16.785310 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.785292    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 13:44:16.785825 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.785808    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 13:44:16.785979 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.785961    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-135.ec2.internal\" not found"
Sep 18 13:44:16.786581 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.786563    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 13:44:16.786581 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.786565    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 13:44:16.786733 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.786600    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 13:44:16.786733 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.786687    2569 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 13:44:16.786733 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.786694    2569 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 13:44:16.787278 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.787262    2569 factory.go:55] Registering systemd factory
Sep 18 13:44:16.787278 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.787279    2569 factory.go:223] Registration of the systemd container factory successfully
Sep 18 13:44:16.787486 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.787474    2569 factory.go:153] Registering CRI-O factory
Sep 18 13:44:16.787486 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.787487    2569 factory.go:223] Registration of the crio container factory successfully
Sep 18 13:44:16.787634 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.787552    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 13:44:16.787634 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.787595    2569 factory.go:103] Registering Raw factory
Sep 18 13:44:16.787634 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.787607    2569 manager.go:1196] Started watching for new ooms in manager
Sep 18 13:44:16.788552 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.788537    2569 manager.go:349] Starting recovery of all containers
Sep 18 13:44:16.789207 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.789183    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tkv7q"
Sep 18 13:44:16.795019 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.794998    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tkv7q"
Sep 18 13:44:16.796893 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.796879    2569 manager.go:354] Recovery completed
Sep 18 13:44:16.798357 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.798327    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 13:44:16.798673 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.798646    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-135.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 13:44:16.803269 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.803256    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:16.805100 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.805084    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:16.805156 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.805115    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:16.805156 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.805128    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:16.805646 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.805628    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 13:44:16.805646 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.805648    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:16.805743 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.805653    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 13:44:16.805743 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.805670    2569 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:44:16.806222 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.806207    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:16.806282 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.806232    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:16.806282 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.806246    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:16.806834 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.806821    2569 policy_none.go:49] "None policy: Start"
Sep 18 13:44:16.806880 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.806837    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 13:44:16.806880 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.806846    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 13:44:16.807848 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.807836    2569 policy_none.go:47] "Start"
Sep 18 13:44:16.850037 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.850020    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.851666    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.851714    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.851734    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.851773    2569 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.851956    2569 manager.go:337] "Starting Device Plugin manager"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.851990    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.852002    2569 server.go:87] "Starting device plugin registration server"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.852220    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.852232    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.852359    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.852421    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.852430    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.852933    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 13:44:16.853440 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.852976    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-135.ec2.internal\" not found"
Sep 18 13:44:16.853986 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.853828    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:16.952681 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.952660    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:16.952767 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.952652    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal"]
Sep 18 13:44:16.952822 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.952813    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:16.953882 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.953867    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:16.953945 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.953895    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:16.953945 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.953905    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:16.953945 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.953922    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-135.ec2.internal"
Sep 18 13:44:16.954488 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.954466    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:16.954579 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.954492    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:16.954579 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.954517    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:16.955810 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.955798    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:16.955960 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.955946    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:16.955996 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.955975    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:16.956537 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.956522    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:16.956615 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.956537    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:16.956615 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.956568    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:16.956615 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.956583    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:16.956615 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.956544    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:16.956799 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.956631    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:16.957813 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.957797    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:16.957895 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.957826    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:44:16.958843 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.958830    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:44:16.958917 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.958856    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:44:16.958917 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.958878    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:44:16.961853 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.961839    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-135.ec2.internal"
Sep 18 13:44:16.961926 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.961858    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-135.ec2.internal\": node \"ip-10-0-143-135.ec2.internal\" not found"
Sep 18 13:44:16.972204 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.972190    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-135.ec2.internal\" not found"
Sep 18 13:44:16.987801 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.987780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4347c5c0037a0a26f8c1c7c2c1baa4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal\" (UID: \"f4347c5c0037a0a26f8c1c7c2c1baa4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:16.987864 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.987803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4347c5c0037a0a26f8c1c7c2c1baa4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal\" (UID: \"f4347c5c0037a0a26f8c1c7c2c1baa4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:16.987864 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:16.987818    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52751f6427256e897ca3f2a8285d32e0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-135.ec2.internal\" (UID: \"52751f6427256e897ca3f2a8285d32e0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:16.993647 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.993634    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-135.ec2.internal\" not found" node="ip-10-0-143-135.ec2.internal"
Sep 18 13:44:16.997797 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:16.997783    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-135.ec2.internal\" not found" node="ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.072670 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.072650    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-135.ec2.internal\" not found"
Sep 18 13:44:17.088403 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.088383    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4347c5c0037a0a26f8c1c7c2c1baa4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal\" (UID: \"f4347c5c0037a0a26f8c1c7c2c1baa4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.088464 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.088407    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4347c5c0037a0a26f8c1c7c2c1baa4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal\" (UID: \"f4347c5c0037a0a26f8c1c7c2c1baa4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.088464 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.088424    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52751f6427256e897ca3f2a8285d32e0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-135.ec2.internal\" (UID: \"52751f6427256e897ca3f2a8285d32e0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.088556 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.088479    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4347c5c0037a0a26f8c1c7c2c1baa4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal\" (UID: \"f4347c5c0037a0a26f8c1c7c2c1baa4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.088556 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.088483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4347c5c0037a0a26f8c1c7c2c1baa4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal\" (UID: \"f4347c5c0037a0a26f8c1c7c2c1baa4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.088556 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.088486    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52751f6427256e897ca3f2a8285d32e0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-135.ec2.internal\" (UID: \"52751f6427256e897ca3f2a8285d32e0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.173723 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.173696    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-135.ec2.internal\" not found"
Sep 18 13:44:17.274440 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.274397    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-135.ec2.internal\" not found"
Sep 18 13:44:17.297907 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.297891    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.302491 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.302371    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.375067 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.375038    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-135.ec2.internal\" not found"
Sep 18 13:44:17.475568 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.475553    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-135.ec2.internal\" not found"
Sep 18 13:44:17.576009 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.575989    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-135.ec2.internal\" not found"
Sep 18 13:44:17.623550 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.623531    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:17.686233 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.686217    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.697157 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.697143    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:44:17.698933 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.698921    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal"
Sep 18 13:44:17.708104 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.708093    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:44:17.730241 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.730227    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 13:44:17.730580 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.730325    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 13:44:17.730580 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.730374    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 13:44:17.764528 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.764485    2569 apiserver.go:52] "Watching apiserver"
Sep 18 13:44:17.784493 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.784473    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 13:44:17.785586 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.785553    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-s5mth","openshift-multus/network-metrics-daemon-xrrx8","openshift-network-diagnostics/network-check-target-xsq2g","kube-system/global-pull-secret-syncer-fkhh5","kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal","openshift-network-operator/iptables-alerter-kbfb9","openshift-ovn-kubernetes/ovnkube-node-wcb2h","kube-system/konnectivity-agent-nvrgd","openshift-cluster-node-tuning-operator/tuned-hztq9","openshift-dns/node-resolver-ww9c2","openshift-image-registry/node-ca-2lqjs","openshift-multus/multus-additional-cni-plugins-jzwlw"]
Sep 18 13:44:17.785695 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.785572    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 13:44:17.786839 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.786818    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:17.788757 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.788733    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.789678 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.789644    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4wv82\""
Sep 18 13:44:17.790003 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.789979    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 13:44:17.790126 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.790112    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 13:44:17.790414 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.790350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:17.790606 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.790580    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:17.791399 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.791381    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:17.791480 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.791466    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:17.791918 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.791891    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:44:17.792020 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.791934    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pj9t4\""
Sep 18 13:44:17.792020 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.791970    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6ddd8f8-7119-4e4a-aaf2-3247b824a726-agent-certs\") pod \"konnectivity-agent-nvrgd\" (UID: \"b6ddd8f8-7119-4e4a-aaf2-3247b824a726\") " pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:17.792020 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792000    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 13:44:17.792020 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792006    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-sysconfig\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792203 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792077    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-sysctl-conf\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792203 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792155    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-kubernetes\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792203 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792179    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-systemd\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792305 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792206    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-host\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792380 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792330    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-modprobe-d\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792472 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792451    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.792472 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-sysctl-d\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792614 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-lib-modules\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792614 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792584    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-var-lib-kubelet\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792710 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792628    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-tmp\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792710 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792668    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgphn\" (UniqueName: \"kubernetes.io/projected/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-kube-api-access-kgphn\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792797 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6ddd8f8-7119-4e4a-aaf2-3247b824a726-konnectivity-ca\") pod \"konnectivity-agent-nvrgd\" (UID: \"b6ddd8f8-7119-4e4a-aaf2-3247b824a726\") " pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:17.792797 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792752    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-run\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792797 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792775    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-sys\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.792915 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.792808    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-tuned\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.793813 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.793795    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.795002 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.794984    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.796047 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.796030    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:17.796363 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.796345    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 13:44:17.796425 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.796389    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 13:44:17.796489 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.796476    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 13:44:17.796561 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.796479    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hhjxb\""
Sep 18 13:44:17.797280 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.797254    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:17.797361 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.797310    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:17.797361 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.797353    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 13:44:17.797438 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.797355    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 13:44:17.797585 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.797572    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 13:44:17.798469 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.798458    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.799652 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.799634    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 13:44:17.799731 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.799659    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 13:44:17.799731 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.799703    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-swhqc\""
Sep 18 13:44:17.799830 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.799740    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:17.800129 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.800113    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 13:44:17.800217 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.800205    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 13:44:17.800612 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.800268    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 13:44:17.800612 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.800296    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 13:44:17.800612 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.800341    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 13:44:17.800612 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.800278    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 13:44:17.800612 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.800388    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hwjhn\""
Sep 18 13:44:17.800612 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.800456    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 13:44:17.800924 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.800908    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:17.801813 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.801790    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zzjhx\""
Sep 18 13:44:17.802026 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.802006    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 13:44:17.802392 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.802110    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 13:44:17.802392 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.802121    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 13:44:17.802392 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.802233    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jc4bs\""
Sep 18 13:44:17.802392 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.802258    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 13:39:18 +0000 UTC" deadline="2028-02-25 08:39:59.980405102 +0000 UTC"
Sep 18 13:44:17.802392 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.802276    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12594h55m42.178132131s"
Sep 18 13:44:17.802820 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.802805    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:44:17.803102 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.803085    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 13:44:17.803178 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.803110    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 13:44:17.803178 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.803156    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-flpdq\""
Sep 18 13:44:17.803693 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.803674    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:44:17.804008 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.803991    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 13:44:17.804094 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.803998    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 13:44:17.804094 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.804074    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xl4df\""
Sep 18 13:44:17.826891 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.826846    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-566wl"
Sep 18 13:44:17.841832 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.841817    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-566wl"
Sep 18 13:44:17.875381 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:17.875351    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4347c5c0037a0a26f8c1c7c2c1baa4c.slice/crio-229bf92e2198886cd4f159264e1151d637de4991768ab4e4eae8f274e769e03c WatchSource:0}: Error finding container 229bf92e2198886cd4f159264e1151d637de4991768ab4e4eae8f274e769e03c: Status 404 returned error can't find the container with id 229bf92e2198886cd4f159264e1151d637de4991768ab4e4eae8f274e769e03c
Sep 18 13:44:17.875578 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:17.875566    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52751f6427256e897ca3f2a8285d32e0.slice/crio-291756c453685460560c13f16b3e10776a3c545486b11cca576d0009cea415c8 WatchSource:0}: Error finding container 291756c453685460560c13f16b3e10776a3c545486b11cca576d0009cea415c8: Status 404 returned error can't find the container with id 291756c453685460560c13f16b3e10776a3c545486b11cca576d0009cea415c8
Sep 18 13:44:17.880287 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.880274    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:44:17.887516 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.887478    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 13:44:17.892976 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.892959    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-env-overrides\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.893055 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.892984    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-kubernetes\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.893055 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893000    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-host\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.893055 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85cv6\" (UniqueName: \"kubernetes.io/projected/742dfd2a-3975-49f5-84f5-3bbf7298b399-kube-api-access-85cv6\") pod \"node-ca-2lqjs\" (UID: \"742dfd2a-3975-49f5-84f5-3bbf7298b399\") " pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:17.893055 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893030    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/202208af-3847-426c-a23e-c5ff1cda0730-cni-binary-copy\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.893055 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893043    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b31645a-f2d0-44ba-aa15-0bc175cb4319-dbus\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:17.893257 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893105    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-kubernetes\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.893257 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893164    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-kubelet\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.893257 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893171    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-host\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.893257 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893188    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93e9ed3e-0558-49bb-8267-118a66dd4c05-host-slash\") pod \"iptables-alerter-kbfb9\" (UID: \"93e9ed3e-0558-49bb-8267-118a66dd4c05\") " pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:17.893257 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-lib-modules\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.893420 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-tmp\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.893420 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893310    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-system-cni-dir\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.893420 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-run-multus-certs\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.893420 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893359    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-sys-fs\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.893420 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893379    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-lib-modules\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.893420 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893383    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-run-ovn\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.893732 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-sys\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.893732 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-sys\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.893732 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893520    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-tuned\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.893732 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893546    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-cnibin\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.893732 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893555    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 13:44:17.893732 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893568    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93e9ed3e-0558-49bb-8267-118a66dd4c05-iptables-alerter-script\") pod \"iptables-alerter-kbfb9\" (UID: \"93e9ed3e-0558-49bb-8267-118a66dd4c05\") " pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:17.893732 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893592    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64pjc\" (UniqueName: \"kubernetes.io/projected/ded613ac-9a58-46d0-9ca6-b66852d872cb-kube-api-access-64pjc\") pod \"node-resolver-ww9c2\" (UID: \"ded613ac-9a58-46d0-9ca6-b66852d872cb\") " pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:17.893732 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-registration-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.893732 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893636    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-device-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-sysctl-conf\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893844    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2406c75d-99cf-4635-a988-6e44c91df885-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893879    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893903    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-run-openvswitch\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893925    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-cni-netd\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893934    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-sysctl-conf\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-systemd\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.893984    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2406c75d-99cf-4635-a988-6e44c91df885-system-cni-dir\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894006    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-etc-selinux\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894008    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-systemd\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894030    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-var-lib-openvswitch\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.894109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894070    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ded613ac-9a58-46d0-9ca6-b66852d872cb-hosts-file\") pod \"node-resolver-ww9c2\" (UID: \"ded613ac-9a58-46d0-9ca6-b66852d872cb\") " pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894116    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxqn\" (UniqueName: \"kubernetes.io/projected/2406c75d-99cf-4635-a988-6e44c91df885-kube-api-access-wvxqn\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-modprobe-d\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-sysctl-d\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894172    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-var-lib-kubelet\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgphn\" (UniqueName: \"kubernetes.io/projected/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-kube-api-access-kgphn\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894215    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-var-lib-cni-bin\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894236    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-systemd-units\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894243    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-modprobe-d\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894254    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljxd\" (UniqueName: \"kubernetes.io/projected/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-kube-api-access-bljxd\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-run\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894305    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2406c75d-99cf-4635-a988-6e44c91df885-cni-binary-copy\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894329    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-slash\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894352    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-ovnkube-config\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ded613ac-9a58-46d0-9ca6-b66852d872cb-tmp-dir\") pod \"node-resolver-ww9c2\" (UID: \"ded613ac-9a58-46d0-9ca6-b66852d872cb\") " pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894395    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-multus-socket-dir-parent\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.894712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894420    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894448    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-ovn-node-metrics-cert\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-var-lib-kubelet\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894526    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/202208af-3847-426c-a23e-c5ff1cda0730-multus-daemon-config\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b31645a-f2d0-44ba-aa15-0bc175cb4319-kubelet-config\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894576    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pf5\" (UniqueName: \"kubernetes.io/projected/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-kube-api-access-b4pf5\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-cni-bin\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-var-lib-kubelet\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894623    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-sysctl-d\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894647    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-run-k8s-cni-cncf-io\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894764    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-run\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894806    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-etc-kubernetes\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894824    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqzt\" (UniqueName: \"kubernetes.io/projected/202208af-3847-426c-a23e-c5ff1cda0730-kube-api-access-mnqzt\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894852    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6ddd8f8-7119-4e4a-aaf2-3247b824a726-konnectivity-ca\") pod \"konnectivity-agent-nvrgd\" (UID: \"b6ddd8f8-7119-4e4a-aaf2-3247b824a726\") " pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2406c75d-99cf-4635-a988-6e44c91df885-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894902    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-multus-cni-dir\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.895462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-sysconfig\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894961    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-run-netns\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894977    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-var-lib-cni-multus\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.894994    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-sysconfig\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-multus-conf-dir\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895094    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895125    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-ovnkube-script-lib\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj2k6\" (UniqueName: \"kubernetes.io/projected/93e9ed3e-0558-49bb-8267-118a66dd4c05-kube-api-access-qj2k6\") pod \"iptables-alerter-kbfb9\" (UID: \"93e9ed3e-0558-49bb-8267-118a66dd4c05\") " pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8qq\" (UniqueName: \"kubernetes.io/projected/b2ce9daf-142e-4306-8a94-939272a094d2-kube-api-access-rj8qq\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895195    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-socket-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895230    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/742dfd2a-3975-49f5-84f5-3bbf7298b399-host\") pod \"node-ca-2lqjs\" (UID: \"742dfd2a-3975-49f5-84f5-3bbf7298b399\") " pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2406c75d-99cf-4635-a988-6e44c91df885-cnibin\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895337    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-hostroot\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895363    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-log-socket\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895386    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-run-ovn-kubernetes\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2406c75d-99cf-4635-a988-6e44c91df885-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.896144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-run-systemd\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.896817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-etc-openvswitch\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.896817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895516    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6ddd8f8-7119-4e4a-aaf2-3247b824a726-agent-certs\") pod \"konnectivity-agent-nvrgd\" (UID: \"b6ddd8f8-7119-4e4a-aaf2-3247b824a726\") " pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:17.896817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895541    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdkjk\" (UniqueName: \"kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk\") pod \"network-check-target-xsq2g\" (UID: \"0105fcdc-d0aa-441c-bb90-83c7419550d6\") " pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:17.896817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/742dfd2a-3975-49f5-84f5-3bbf7298b399-serviceca\") pod \"node-ca-2lqjs\" (UID: \"742dfd2a-3975-49f5-84f5-3bbf7298b399\") " pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:17.896817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-run-netns\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.896817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895570    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6ddd8f8-7119-4e4a-aaf2-3247b824a726-konnectivity-ca\") pod \"konnectivity-agent-nvrgd\" (UID: \"b6ddd8f8-7119-4e4a-aaf2-3247b824a726\") " pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:17.896817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.895620    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-node-log\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.897378 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.897352    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-tmp\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.897469 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.897401    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-etc-tuned\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.898132 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.898112    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6ddd8f8-7119-4e4a-aaf2-3247b824a726-agent-certs\") pod \"konnectivity-agent-nvrgd\" (UID: \"b6ddd8f8-7119-4e4a-aaf2-3247b824a726\") " pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:17.902464 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.902438    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgphn\" (UniqueName: \"kubernetes.io/projected/48b4f338-f860-49eb-b16a-cb7d4e9fc63f-kube-api-access-kgphn\") pod \"tuned-hztq9\" (UID: \"48b4f338-f860-49eb-b16a-cb7d4e9fc63f\") " pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:17.996305 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996279    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxqn\" (UniqueName: \"kubernetes.io/projected/2406c75d-99cf-4635-a988-6e44c91df885-kube-api-access-wvxqn\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.996430 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996309    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-var-lib-cni-bin\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.996430 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996328    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-systemd-units\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.996430 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljxd\" (UniqueName: \"kubernetes.io/projected/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-kube-api-access-bljxd\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.996430 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2406c75d-99cf-4635-a988-6e44c91df885-cni-binary-copy\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.996654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996429    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-var-lib-cni-bin\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.996654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996460    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-systemd-units\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.996654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-slash\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.996654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-ovnkube-config\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.996654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ded613ac-9a58-46d0-9ca6-b66852d872cb-tmp-dir\") pod \"node-resolver-ww9c2\" (UID: \"ded613ac-9a58-46d0-9ca6-b66852d872cb\") " pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:17.996654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-multus-socket-dir-parent\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.996654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996601    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.996654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996575    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-slash\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.996654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-ovn-node-metrics-cert\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996663    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-var-lib-kubelet\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996693    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/202208af-3847-426c-a23e-c5ff1cda0730-multus-daemon-config\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b31645a-f2d0-44ba-aa15-0bc175cb4319-kubelet-config\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pf5\" (UniqueName: \"kubernetes.io/projected/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-kube-api-access-b4pf5\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996763    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-cni-bin\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996790    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996813    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-run-k8s-cni-cncf-io\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996835    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-etc-kubernetes\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996856    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqzt\" (UniqueName: \"kubernetes.io/projected/202208af-3847-426c-a23e-c5ff1cda0730-kube-api-access-mnqzt\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2406c75d-99cf-4635-a988-6e44c91df885-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996907    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-multus-cni-dir\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-run-netns\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-var-lib-cni-multus\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996972    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ded613ac-9a58-46d0-9ca6-b66852d872cb-tmp-dir\") pod \"node-resolver-ww9c2\" (UID: \"ded613ac-9a58-46d0-9ca6-b66852d872cb\") " pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996974    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-var-lib-kubelet\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996979    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-multus-conf-dir\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997061 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996761    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997079    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-run-k8s-cni-cncf-io\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997079    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-multus-conf-dir\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997109    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2406c75d-99cf-4635-a988-6e44c91df885-cni-binary-copy\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997118    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-etc-kubernetes\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997143    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-multus-cni-dir\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.997149    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997152    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b31645a-f2d0-44ba-aa15-0bc175cb4319-kubelet-config\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997202    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-cni-bin\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.996662    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-multus-socket-dir-parent\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.997208    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.997230    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret podName:4b31645a-f2d0-44ba-aa15-0bc175cb4319 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:18.497188696 +0000 UTC m=+2.027214262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret") pod "global-pull-secret-syncer-fkhh5" (UID: "4b31645a-f2d0-44ba-aa15-0bc175cb4319") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997302    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/202208af-3847-426c-a23e-c5ff1cda0730-multus-daemon-config\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997335    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-run-netns\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-ovnkube-script-lib\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:17.997389    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs podName:b2ce9daf-142e-4306-8a94-939272a094d2 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:18.497371285 +0000 UTC m=+2.027396858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs") pod "network-metrics-daemon-xrrx8" (UID: "b2ce9daf-142e-4306-8a94-939272a094d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:17.997861 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997409    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj2k6\" (UniqueName: \"kubernetes.io/projected/93e9ed3e-0558-49bb-8267-118a66dd4c05-kube-api-access-qj2k6\") pod \"iptables-alerter-kbfb9\" (UID: \"93e9ed3e-0558-49bb-8267-118a66dd4c05\") " pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997437    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8qq\" (UniqueName: \"kubernetes.io/projected/b2ce9daf-142e-4306-8a94-939272a094d2-kube-api-access-rj8qq\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-socket-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997488    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/742dfd2a-3975-49f5-84f5-3bbf7298b399-host\") pod \"node-ca-2lqjs\" (UID: \"742dfd2a-3975-49f5-84f5-3bbf7298b399\") " pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997530    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2406c75d-99cf-4635-a988-6e44c91df885-cnibin\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997545    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-var-lib-cni-multus\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997554    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-hostroot\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997588    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-log-socket\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997588    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-hostroot\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-run-ovn-kubernetes\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2406c75d-99cf-4635-a988-6e44c91df885-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997680    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-run-systemd\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-etc-openvswitch\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997735    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkjk\" (UniqueName: \"kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk\") pod \"network-check-target-xsq2g\" (UID: \"0105fcdc-d0aa-441c-bb90-83c7419550d6\") " pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997758    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/742dfd2a-3975-49f5-84f5-3bbf7298b399-serviceca\") pod \"node-ca-2lqjs\" (UID: \"742dfd2a-3975-49f5-84f5-3bbf7298b399\") " pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997782    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-run-netns\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997792    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-run-systemd\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.998953 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997801    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-node-log\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-env-overrides\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997845    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-ovnkube-script-lib\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997865    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85cv6\" (UniqueName: \"kubernetes.io/projected/742dfd2a-3975-49f5-84f5-3bbf7298b399-kube-api-access-85cv6\") pod \"node-ca-2lqjs\" (UID: \"742dfd2a-3975-49f5-84f5-3bbf7298b399\") " pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997888    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-run-netns\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/202208af-3847-426c-a23e-c5ff1cda0730-cni-binary-copy\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b31645a-f2d0-44ba-aa15-0bc175cb4319-dbus\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997682    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-log-socket\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997961    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2406c75d-99cf-4635-a988-6e44c91df885-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997654    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-run-ovn-kubernetes\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997998    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-etc-openvswitch\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.997758    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-socket-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998041    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-node-log\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998092    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-kubelet\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998116    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93e9ed3e-0558-49bb-8267-118a66dd4c05-host-slash\") pod \"iptables-alerter-kbfb9\" (UID: \"93e9ed3e-0558-49bb-8267-118a66dd4c05\") " pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998154    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-system-cni-dir\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-run-multus-certs\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-sys-fs\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:17.999758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/742dfd2a-3975-49f5-84f5-3bbf7298b399-serviceca\") pod \"node-ca-2lqjs\" (UID: \"742dfd2a-3975-49f5-84f5-3bbf7298b399\") " pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998274    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-host-run-multus-certs\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998277    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93e9ed3e-0558-49bb-8267-118a66dd4c05-host-slash\") pod \"iptables-alerter-kbfb9\" (UID: \"93e9ed3e-0558-49bb-8267-118a66dd4c05\") " pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998285    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2406c75d-99cf-4635-a988-6e44c91df885-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-run-ovn\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-sys-fs\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b31645a-f2d0-44ba-aa15-0bc175cb4319-dbus\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998334    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-kubelet\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/742dfd2a-3975-49f5-84f5-3bbf7298b399-host\") pod \"node-ca-2lqjs\" (UID: \"742dfd2a-3975-49f5-84f5-3bbf7298b399\") " pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998373    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-cnibin\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-env-overrides\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998403    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93e9ed3e-0558-49bb-8267-118a66dd4c05-iptables-alerter-script\") pod \"iptables-alerter-kbfb9\" (UID: \"93e9ed3e-0558-49bb-8267-118a66dd4c05\") " pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998435    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64pjc\" (UniqueName: \"kubernetes.io/projected/ded613ac-9a58-46d0-9ca6-b66852d872cb-kube-api-access-64pjc\") pod \"node-resolver-ww9c2\" (UID: \"ded613ac-9a58-46d0-9ca6-b66852d872cb\") " pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-registration-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998463    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-run-ovn\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998438    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2406c75d-99cf-4635-a988-6e44c91df885-cnibin\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-system-cni-dir\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:18.000335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998516    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-device-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998536    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-registration-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998545    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/202208af-3847-426c-a23e-c5ff1cda0730-cnibin\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2406c75d-99cf-4635-a988-6e44c91df885-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998591    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-device-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-run-openvswitch\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998733    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-cni-netd\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998778    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-run-openvswitch\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998776    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2406c75d-99cf-4635-a988-6e44c91df885-system-cni-dir\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998843    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998879    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-etc-selinux\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998899    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/202208af-3847-426c-a23e-c5ff1cda0730-cni-binary-copy\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998907    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-var-lib-openvswitch\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998906    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-ovnkube-config\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.998935    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ded613ac-9a58-46d0-9ca6-b66852d872cb-hosts-file\") pod \"node-resolver-ww9c2\" (UID: \"ded613ac-9a58-46d0-9ca6-b66852d872cb\") " pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.999001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-host-cni-netd\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.000812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.999001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2406c75d-99cf-4635-a988-6e44c91df885-system-cni-dir\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:18.001254 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.999021    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-etc-selinux\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.001254 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.999032    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2406c75d-99cf-4635-a988-6e44c91df885-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:18.001254 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.999042    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ded613ac-9a58-46d0-9ca6-b66852d872cb-hosts-file\") pod \"node-resolver-ww9c2\" (UID: \"ded613ac-9a58-46d0-9ca6-b66852d872cb\") " pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:18.001254 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.999051    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-var-lib-openvswitch\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.001254 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:17.999030    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93e9ed3e-0558-49bb-8267-118a66dd4c05-iptables-alerter-script\") pod \"iptables-alerter-kbfb9\" (UID: \"93e9ed3e-0558-49bb-8267-118a66dd4c05\") " pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:18.001254 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.000228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-ovn-node-metrics-cert\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.007772 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.007735    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pf5\" (UniqueName: \"kubernetes.io/projected/12b683a1-3f2d-4b34-97e6-d76ed9d00fef-kube-api-access-b4pf5\") pod \"aws-ebs-csi-driver-node-f54l7\" (UID: \"12b683a1-3f2d-4b34-97e6-d76ed9d00fef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.008090 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.008069    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqzt\" (UniqueName: \"kubernetes.io/projected/202208af-3847-426c-a23e-c5ff1cda0730-kube-api-access-mnqzt\") pod \"multus-s5mth\" (UID: \"202208af-3847-426c-a23e-c5ff1cda0730\") " pod="openshift-multus/multus-s5mth"
Sep 18 13:44:18.008603 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.008580    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxqn\" (UniqueName: \"kubernetes.io/projected/2406c75d-99cf-4635-a988-6e44c91df885-kube-api-access-wvxqn\") pod \"multus-additional-cni-plugins-jzwlw\" (UID: \"2406c75d-99cf-4635-a988-6e44c91df885\") " pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:18.008790 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.008774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljxd\" (UniqueName: \"kubernetes.io/projected/2fbbca64-d3f9-42c6-8175-94b778fbdcb7-kube-api-access-bljxd\") pod \"ovnkube-node-wcb2h\" (UID: \"2fbbca64-d3f9-42c6-8175-94b778fbdcb7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.009630 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.009573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj2k6\" (UniqueName: \"kubernetes.io/projected/93e9ed3e-0558-49bb-8267-118a66dd4c05-kube-api-access-qj2k6\") pod \"iptables-alerter-kbfb9\" (UID: \"93e9ed3e-0558-49bb-8267-118a66dd4c05\") " pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:18.010253 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.010235    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8qq\" (UniqueName: \"kubernetes.io/projected/b2ce9daf-142e-4306-8a94-939272a094d2-kube-api-access-rj8qq\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:18.010338 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.010329    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:18.010399 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.010340    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:18.010399 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.010349    2569 projected.go:196] Error preparing data for projected volume kube-api-access-xdkjk for pod openshift-network-diagnostics/network-check-target-xsq2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:18.010399 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.010397    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk podName:0105fcdc-d0aa-441c-bb90-83c7419550d6 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:18.51038313 +0000 UTC m=+2.040408690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xdkjk" (UniqueName: "kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk") pod "network-check-target-xsq2g" (UID: "0105fcdc-d0aa-441c-bb90-83c7419550d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:18.011191 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.011173    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64pjc\" (UniqueName: \"kubernetes.io/projected/ded613ac-9a58-46d0-9ca6-b66852d872cb-kube-api-access-64pjc\") pod \"node-resolver-ww9c2\" (UID: \"ded613ac-9a58-46d0-9ca6-b66852d872cb\") " pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:18.011535 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.011514    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85cv6\" (UniqueName: \"kubernetes.io/projected/742dfd2a-3975-49f5-84f5-3bbf7298b399-kube-api-access-85cv6\") pod \"node-ca-2lqjs\" (UID: \"742dfd2a-3975-49f5-84f5-3bbf7298b399\") " pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:18.108550 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.108471    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:18.116698 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:18.116678    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ddd8f8_7119_4e4a_aaf2_3247b824a726.slice/crio-a9255c2869a2f59d15b333e093a677dcac32893b0dcf627bd30eca0f76df8d48 WatchSource:0}: Error finding container a9255c2869a2f59d15b333e093a677dcac32893b0dcf627bd30eca0f76df8d48: Status 404 returned error can't find the container with id a9255c2869a2f59d15b333e093a677dcac32893b0dcf627bd30eca0f76df8d48
Sep 18 13:44:18.132530 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.132493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hztq9"
Sep 18 13:44:18.139809 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:18.139790    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b4f338_f860_49eb_b16a_cb7d4e9fc63f.slice/crio-95de885f822067e2149c4fd6a1313f550d63e5d76b70a06970cb2eb3fbdea0c3 WatchSource:0}: Error finding container 95de885f822067e2149c4fd6a1313f550d63e5d76b70a06970cb2eb3fbdea0c3: Status 404 returned error can't find the container with id 95de885f822067e2149c4fd6a1313f550d63e5d76b70a06970cb2eb3fbdea0c3
Sep 18 13:44:18.152184 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.152162    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jzwlw"
Sep 18 13:44:18.160081 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:18.160062    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2406c75d_99cf_4635_a988_6e44c91df885.slice/crio-4d3da2d341cdb2d56abe95447a7d120a7dd8d0c322917719bdabfd7ce2a6af46 WatchSource:0}: Error finding container 4d3da2d341cdb2d56abe95447a7d120a7dd8d0c322917719bdabfd7ce2a6af46: Status 404 returned error can't find the container with id 4d3da2d341cdb2d56abe95447a7d120a7dd8d0c322917719bdabfd7ce2a6af46
Sep 18 13:44:18.175677 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.175646    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7"
Sep 18 13:44:18.181482 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.181465    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:18.182210 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:18.182188    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b683a1_3f2d_4b34_97e6_d76ed9d00fef.slice/crio-1235649788f8cd45cd4e29e67286435eeaf1eceb09641af38750025b7a7e535e WatchSource:0}: Error finding container 1235649788f8cd45cd4e29e67286435eeaf1eceb09641af38750025b7a7e535e: Status 404 returned error can't find the container with id 1235649788f8cd45cd4e29e67286435eeaf1eceb09641af38750025b7a7e535e
Sep 18 13:44:18.187404 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.187389    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2lqjs"
Sep 18 13:44:18.187835 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.187817    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:18.188071 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:18.188055    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbbca64_d3f9_42c6_8175_94b778fbdcb7.slice/crio-0fc1e6f99256ecf8d281d98a6353363f14259bf9c2ae13148a6af5b9fcf9f02b WatchSource:0}: Error finding container 0fc1e6f99256ecf8d281d98a6353363f14259bf9c2ae13148a6af5b9fcf9f02b: Status 404 returned error can't find the container with id 0fc1e6f99256ecf8d281d98a6353363f14259bf9c2ae13148a6af5b9fcf9f02b
Sep 18 13:44:18.193187 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.193161    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5mth"
Sep 18 13:44:18.194560 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:18.194539    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742dfd2a_3975_49f5_84f5_3bbf7298b399.slice/crio-b7d263f13abccda9941736cda3935433b6d41452e73269cff2d310b3d58276b0 WatchSource:0}: Error finding container b7d263f13abccda9941736cda3935433b6d41452e73269cff2d310b3d58276b0: Status 404 returned error can't find the container with id b7d263f13abccda9941736cda3935433b6d41452e73269cff2d310b3d58276b0
Sep 18 13:44:18.199863 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:18.199846    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202208af_3847_426c_a23e_c5ff1cda0730.slice/crio-47b6a4ab6ad6178f6433a1e122a56a77172f4add2008f966c1d01769fc1c5b9a WatchSource:0}: Error finding container 47b6a4ab6ad6178f6433a1e122a56a77172f4add2008f966c1d01769fc1c5b9a: Status 404 returned error can't find the container with id 47b6a4ab6ad6178f6433a1e122a56a77172f4add2008f966c1d01769fc1c5b9a
Sep 18 13:44:18.200386 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.200370    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kbfb9"
Sep 18 13:44:18.204564 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.204548    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ww9c2"
Sep 18 13:44:18.207284 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:18.207266    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e9ed3e_0558_49bb_8267_118a66dd4c05.slice/crio-c30b446ae94851aaaca2f4f397e19a2547fa7a26d199681d4279dcbd9c251ac9 WatchSource:0}: Error finding container c30b446ae94851aaaca2f4f397e19a2547fa7a26d199681d4279dcbd9c251ac9: Status 404 returned error can't find the container with id c30b446ae94851aaaca2f4f397e19a2547fa7a26d199681d4279dcbd9c251ac9
Sep 18 13:44:18.211802 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:18.211784    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded613ac_9a58_46d0_9ca6_b66852d872cb.slice/crio-305fcc76eb037ebd6a02a6c6c574d78f45f794a04eab3099dcb493eebabcc2e9 WatchSource:0}: Error finding container 305fcc76eb037ebd6a02a6c6c574d78f45f794a04eab3099dcb493eebabcc2e9: Status 404 returned error can't find the container with id 305fcc76eb037ebd6a02a6c6c574d78f45f794a04eab3099dcb493eebabcc2e9
Sep 18 13:44:18.252800 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.252777    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:18.501586 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.501490    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:18.501586 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.501561    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:18.501789 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.501697    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:18.501789 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.501697    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:18.501789 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.501751    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret podName:4b31645a-f2d0-44ba-aa15-0bc175cb4319 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:19.501735087 +0000 UTC m=+3.031760647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret") pod "global-pull-secret-syncer-fkhh5" (UID: "4b31645a-f2d0-44ba-aa15-0bc175cb4319") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:18.501789 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.501766    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs podName:b2ce9daf-142e-4306-8a94-939272a094d2 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:19.501758779 +0000 UTC m=+3.031784326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs") pod "network-metrics-daemon-xrrx8" (UID: "b2ce9daf-142e-4306-8a94-939272a094d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:18.602995 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:18.602954    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkjk\" (UniqueName: \"kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk\") pod \"network-check-target-xsq2g\" (UID: \"0105fcdc-d0aa-441c-bb90-83c7419550d6\") " pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:18.603172 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.603109    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:18.603172 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.603131    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:18.603172 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.603143    2569 projected.go:196] Error preparing data for projected volume kube-api-access-xdkjk for pod openshift-network-diagnostics/network-check-target-xsq2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:18.603325 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:18.603197    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk podName:0105fcdc-d0aa-441c-bb90-83c7419550d6 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:19.603179947 +0000 UTC m=+3.133205514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdkjk" (UniqueName: "kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk") pod "network-check-target-xsq2g" (UID: "0105fcdc-d0aa-441c-bb90-83c7419550d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:20.471636 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.471439    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:39:19 +0000 UTC" deadline="2028-04-08 15:41:48.391637369 +0000 UTC"
Sep 18 13:44:20.471636 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.471538    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13633h57m27.920104524s"
Sep 18 13:44:20.481999 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.481973    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:20.482335 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:20.482205    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:20.510207 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.510149    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ww9c2" event={"ID":"ded613ac-9a58-46d0-9ca6-b66852d872cb","Type":"ContainerStarted","Data":"305fcc76eb037ebd6a02a6c6c574d78f45f794a04eab3099dcb493eebabcc2e9"}
Sep 18 13:44:20.532276 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.532173    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5mth" event={"ID":"202208af-3847-426c-a23e-c5ff1cda0730","Type":"ContainerStarted","Data":"47b6a4ab6ad6178f6433a1e122a56a77172f4add2008f966c1d01769fc1c5b9a"}
Sep 18 13:44:20.537005 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.536973    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2lqjs" event={"ID":"742dfd2a-3975-49f5-84f5-3bbf7298b399","Type":"ContainerStarted","Data":"b7d263f13abccda9941736cda3935433b6d41452e73269cff2d310b3d58276b0"}
Sep 18 13:44:20.543598 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.543571    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7" event={"ID":"12b683a1-3f2d-4b34-97e6-d76ed9d00fef","Type":"ContainerStarted","Data":"1235649788f8cd45cd4e29e67286435eeaf1eceb09641af38750025b7a7e535e"}
Sep 18 13:44:20.550461 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.550435    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzwlw" event={"ID":"2406c75d-99cf-4635-a988-6e44c91df885","Type":"ContainerStarted","Data":"4d3da2d341cdb2d56abe95447a7d120a7dd8d0c322917719bdabfd7ce2a6af46"}
Sep 18 13:44:20.553108 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.553058    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hztq9" event={"ID":"48b4f338-f860-49eb-b16a-cb7d4e9fc63f","Type":"ContainerStarted","Data":"95de885f822067e2149c4fd6a1313f550d63e5d76b70a06970cb2eb3fbdea0c3"}
Sep 18 13:44:20.559607 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.559583    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nvrgd" event={"ID":"b6ddd8f8-7119-4e4a-aaf2-3247b824a726","Type":"ContainerStarted","Data":"a9255c2869a2f59d15b333e093a677dcac32893b0dcf627bd30eca0f76df8d48"}
Sep 18 13:44:20.566276 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.566250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal" event={"ID":"52751f6427256e897ca3f2a8285d32e0","Type":"ContainerStarted","Data":"291756c453685460560c13f16b3e10776a3c545486b11cca576d0009cea415c8"}
Sep 18 13:44:20.574759 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.574731    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kbfb9" event={"ID":"93e9ed3e-0558-49bb-8267-118a66dd4c05","Type":"ContainerStarted","Data":"c30b446ae94851aaaca2f4f397e19a2547fa7a26d199681d4279dcbd9c251ac9"}
Sep 18 13:44:20.587275 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.587247    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" event={"ID":"2fbbca64-d3f9-42c6-8175-94b778fbdcb7","Type":"ContainerStarted","Data":"0fc1e6f99256ecf8d281d98a6353363f14259bf9c2ae13148a6af5b9fcf9f02b"}
Sep 18 13:44:20.602430 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.602398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal" event={"ID":"f4347c5c0037a0a26f8c1c7c2c1baa4c","Type":"ContainerStarted","Data":"229bf92e2198886cd4f159264e1151d637de4991768ab4e4eae8f274e769e03c"}
Sep 18 13:44:20.733710 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:20.733638    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:21.140105 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:21.140011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:21.140105 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:21.140079    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:21.140328 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:21.140221    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:21.140328 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:21.140284    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret podName:4b31645a-f2d0-44ba-aa15-0bc175cb4319 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:23.140267109 +0000 UTC m=+5.042723290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret") pod "global-pull-secret-syncer-fkhh5" (UID: "4b31645a-f2d0-44ba-aa15-0bc175cb4319") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:21.140716 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:21.140695    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:21.140814 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:21.140751    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs podName:b2ce9daf-142e-4306-8a94-939272a094d2 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:23.14073525 +0000 UTC m=+5.043191433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs") pod "network-metrics-daemon-xrrx8" (UID: "b2ce9daf-142e-4306-8a94-939272a094d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:21.241094 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:21.240976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkjk\" (UniqueName: \"kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk\") pod \"network-check-target-xsq2g\" (UID: \"0105fcdc-d0aa-441c-bb90-83c7419550d6\") " pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:21.241272 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:21.241209    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:21.241272 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:21.241230    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:21.241272 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:21.241242    2569 projected.go:196] Error preparing data for projected volume kube-api-access-xdkjk for pod openshift-network-diagnostics/network-check-target-xsq2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:21.241419 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:21.241301    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk podName:0105fcdc-d0aa-441c-bb90-83c7419550d6 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:23.241283528 +0000 UTC m=+5.143739714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdkjk" (UniqueName: "kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk") pod "network-check-target-xsq2g" (UID: "0105fcdc-d0aa-441c-bb90-83c7419550d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:21.417176 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:21.417098    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:44:21.472187 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:21.472148    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:39:19 +0000 UTC" deadline="2028-05-23 01:37:31.857245802 +0000 UTC"
Sep 18 13:44:21.472187 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:21.472186    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14699h53m10.38506408s"
Sep 18 13:44:21.480581 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:21.480514    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:21.480770 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:21.480637    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:21.481028 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:21.481008    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:21.481183 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:21.481136    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:22.480519 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:22.480486    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:22.480940 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:22.480616    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:23.160910 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:23.160871    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:23.161114 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:23.160989    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:23.161188 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:23.161141    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:23.161235 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:23.161200    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs podName:b2ce9daf-142e-4306-8a94-939272a094d2 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:27.16118264 +0000 UTC m=+9.063638811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs") pod "network-metrics-daemon-xrrx8" (UID: "b2ce9daf-142e-4306-8a94-939272a094d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:23.161291 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:23.161265    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:23.161342 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:23.161297    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret podName:4b31645a-f2d0-44ba-aa15-0bc175cb4319 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:27.161286681 +0000 UTC m=+9.063742849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret") pod "global-pull-secret-syncer-fkhh5" (UID: "4b31645a-f2d0-44ba-aa15-0bc175cb4319") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:23.261722 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:23.261661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkjk\" (UniqueName: \"kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk\") pod \"network-check-target-xsq2g\" (UID: \"0105fcdc-d0aa-441c-bb90-83c7419550d6\") " pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:23.261891 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:23.261851    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:23.261891 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:23.261875    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:23.261891 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:23.261888    2569 projected.go:196] Error preparing data for projected volume kube-api-access-xdkjk for pod openshift-network-diagnostics/network-check-target-xsq2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:23.262045 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:23.261953    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk podName:0105fcdc-d0aa-441c-bb90-83c7419550d6 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:27.261935693 +0000 UTC m=+9.164391859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdkjk" (UniqueName: "kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk") pod "network-check-target-xsq2g" (UID: "0105fcdc-d0aa-441c-bb90-83c7419550d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:23.479802 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:23.479770    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:23.479969 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:23.479909    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:23.480349 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:23.480329    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:23.480472 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:23.480426    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:24.482809 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:24.482318    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:24.482809 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:24.482450    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:25.480242 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:25.480206    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:25.480428 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:25.480327    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:25.480677 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:25.480655    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:25.480773 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:25.480755    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:26.480151 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:26.480107    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:26.480588 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:26.480254    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:27.196497 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:27.196459    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:27.196669 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:27.196509    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:27.196669 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:27.196599    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:27.196669 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:27.196625    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:27.196669 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:27.196657    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs podName:b2ce9daf-142e-4306-8a94-939272a094d2 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:35.196641842 +0000 UTC m=+17.099098022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs") pod "network-metrics-daemon-xrrx8" (UID: "b2ce9daf-142e-4306-8a94-939272a094d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:27.196874 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:27.196678    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret podName:4b31645a-f2d0-44ba-aa15-0bc175cb4319 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:35.196662399 +0000 UTC m=+17.099118569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret") pod "global-pull-secret-syncer-fkhh5" (UID: "4b31645a-f2d0-44ba-aa15-0bc175cb4319") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:27.297230 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:27.297194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkjk\" (UniqueName: \"kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk\") pod \"network-check-target-xsq2g\" (UID: \"0105fcdc-d0aa-441c-bb90-83c7419550d6\") " pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:27.297400 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:27.297379    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:27.297474 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:27.297412    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:27.297474 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:27.297425    2569 projected.go:196] Error preparing data for projected volume kube-api-access-xdkjk for pod openshift-network-diagnostics/network-check-target-xsq2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:27.297585 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:27.297476    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk podName:0105fcdc-d0aa-441c-bb90-83c7419550d6 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:35.297460323 +0000 UTC m=+17.199916492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdkjk" (UniqueName: "kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk") pod "network-check-target-xsq2g" (UID: "0105fcdc-d0aa-441c-bb90-83c7419550d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:27.480456 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:27.480381    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:27.480843 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:27.480395    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:27.480843 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:27.480529    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:27.480843 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:27.480591    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:28.483100 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:28.481605    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:28.483100 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:28.481736    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:29.479975 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:29.479941    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:29.479975 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:29.479973    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:29.480233 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:29.480063    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:29.480545 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:29.480503    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:30.479692 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:30.479656    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:30.480152 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:30.479787    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:31.480137 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:31.480104    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:31.480537 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:31.480148    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:31.480537 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:31.480220    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:31.480537 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:31.480336    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:32.479652 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:32.479618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:32.479818 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:32.479722    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:33.479983 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:33.479953    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:33.480387 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:33.480050    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:33.480387 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:33.480105    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:33.480387 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:33.480191    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:34.479768 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:34.479740    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:34.479914 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:34.479829    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:35.258082 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:35.258029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:35.258563 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:35.258094    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:35.258563 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:35.258175    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:35.258563 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:35.258187    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:35.258563 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:35.258254    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs podName:b2ce9daf-142e-4306-8a94-939272a094d2 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:51.258233691 +0000 UTC m=+33.160689894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs") pod "network-metrics-daemon-xrrx8" (UID: "b2ce9daf-142e-4306-8a94-939272a094d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:35.258563 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:35.258275    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret podName:4b31645a-f2d0-44ba-aa15-0bc175cb4319 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:51.258266034 +0000 UTC m=+33.160722211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret") pod "global-pull-secret-syncer-fkhh5" (UID: "4b31645a-f2d0-44ba-aa15-0bc175cb4319") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:35.358767 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:35.358736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkjk\" (UniqueName: \"kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk\") pod \"network-check-target-xsq2g\" (UID: \"0105fcdc-d0aa-441c-bb90-83c7419550d6\") " pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:35.358915 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:35.358897    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:35.358915 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:35.358913    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:35.358994 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:35.358922    2569 projected.go:196] Error preparing data for projected volume kube-api-access-xdkjk for pod openshift-network-diagnostics/network-check-target-xsq2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:35.358994 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:35.358971    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk podName:0105fcdc-d0aa-441c-bb90-83c7419550d6 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:51.358955372 +0000 UTC m=+33.261411537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdkjk" (UniqueName: "kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk") pod "network-check-target-xsq2g" (UID: "0105fcdc-d0aa-441c-bb90-83c7419550d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:35.480440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:35.480410    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:35.480585 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:35.480424    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:35.480585 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:35.480527    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:35.480695 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:35.480606    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:36.480019 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:36.479989    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:36.480455 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:36.480120    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:37.479798 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:37.479767    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:37.479932 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:37.479767    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:37.479932 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:37.479869    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:37.480011 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:37.479933    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:38.480438 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.480305    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:38.480720 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:38.480502    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:38.653362 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.653126    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:44:38.654817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.654685    2569 generic.go:356] "Generic (PLEG): container finished" podID="2fbbca64-d3f9-42c6-8175-94b778fbdcb7" containerID="57ef3ffc25d6df249978e3dcbcc84617c8cc30ae556b1b33c1d2b978e3bf695b" exitCode=1
Sep 18 13:44:38.654817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.654808    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" event={"ID":"2fbbca64-d3f9-42c6-8175-94b778fbdcb7","Type":"ContainerDied","Data":"57ef3ffc25d6df249978e3dcbcc84617c8cc30ae556b1b33c1d2b978e3bf695b"}
Sep 18 13:44:38.654958 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.654931    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" event={"ID":"2fbbca64-d3f9-42c6-8175-94b778fbdcb7","Type":"ContainerStarted","Data":"b0f05583e6dd1750f73bcae0e88e4a32d40cd73bc45080783d8761dedf70e64c"}
Sep 18 13:44:38.658533 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.658443    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5mth" event={"ID":"202208af-3847-426c-a23e-c5ff1cda0730","Type":"ContainerStarted","Data":"8921691b63bf943f5df302e74e9ceedb0894a34635f22da15b24f8348bf23deb"}
Sep 18 13:44:38.664308 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.664255    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hztq9" event={"ID":"48b4f338-f860-49eb-b16a-cb7d4e9fc63f","Type":"ContainerStarted","Data":"727ccfafb6c3cdcd08b3fd3eb7ff80a9907ffdb56dd8cfe2e5de7290798f91d1"}
Sep 18 13:44:38.673579 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.673535    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal" event={"ID":"52751f6427256e897ca3f2a8285d32e0","Type":"ContainerStarted","Data":"af480d5b5218cde0c5e451595352a11e4ab53d749a0a0829081ef491a1b24401"}
Sep 18 13:44:38.679361 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.679310    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s5mth" podStartSLOduration=2.368666388 podStartE2EDuration="20.679295621s" totalImagesPullingTime="18.310629233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:38.678957527 +0000 UTC m=+20.581413713" watchObservedRunningTime="2026-09-18 13:44:38.679295621 +0000 UTC m=+20.581751806"
Sep 18 13:44:38.700791 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.700704    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hztq9" podStartSLOduration=2.329909131 podStartE2EDuration="20.700689096s" totalImagesPullingTime="18.370779965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:38.699901665 +0000 UTC m=+20.602357849" watchObservedRunningTime="2026-09-18 13:44:38.700689096 +0000 UTC m=+20.603145280"
Sep 18 13:44:38.717550 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:38.717501    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-135.ec2.internal" podStartSLOduration=19.717483063 podStartE2EDuration="19.717483063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:38.717239468 +0000 UTC m=+20.619695659" watchObservedRunningTime="2026-09-18 13:44:38.717483063 +0000 UTC m=+20.619939248"
Sep 18 13:44:39.480360 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.480338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:39.480485 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:39.480431    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:39.481180 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.480490    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:39.481180 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:39.480571    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:39.557769 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.557589    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 13:44:39.679744 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.679714    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ww9c2" event={"ID":"ded613ac-9a58-46d0-9ca6-b66852d872cb","Type":"ContainerStarted","Data":"efcef84bf192aa377ef1334c07e4e75d2caf6d61d983a8a21497ae76a6865429"}
Sep 18 13:44:39.681217 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.681194    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2lqjs" event={"ID":"742dfd2a-3975-49f5-84f5-3bbf7298b399","Type":"ContainerStarted","Data":"14f858afd8ec4da3efe5eeef28e6e3afba12ecf4bc49dbce349b3eff46d841e8"}
Sep 18 13:44:39.682908 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.682884    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7" event={"ID":"12b683a1-3f2d-4b34-97e6-d76ed9d00fef","Type":"ContainerStarted","Data":"27e421c6844515e4739d93b4ca3ccdd1bcbe91e6d4bbd4a68de9a8065cfe932e"}
Sep 18 13:44:39.682984 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.682931    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7" event={"ID":"12b683a1-3f2d-4b34-97e6-d76ed9d00fef","Type":"ContainerStarted","Data":"2f00ea934ac0e13b614cd976bbfaa0c8163499ebfc018bbd84a6cdf331b325aa"}
Sep 18 13:44:39.684206 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.684187    2569 generic.go:356] "Generic (PLEG): container finished" podID="2406c75d-99cf-4635-a988-6e44c91df885" containerID="84c381ca9253d3c3ecce00b2bb2d932da5463a7a04b21d0a8dc05bb5daa9643d" exitCode=0
Sep 18 13:44:39.684282 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.684250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzwlw" event={"ID":"2406c75d-99cf-4635-a988-6e44c91df885","Type":"ContainerDied","Data":"84c381ca9253d3c3ecce00b2bb2d932da5463a7a04b21d0a8dc05bb5daa9643d"}
Sep 18 13:44:39.685686 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.685655    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nvrgd" event={"ID":"b6ddd8f8-7119-4e4a-aaf2-3247b824a726","Type":"ContainerStarted","Data":"1bc2ba1db385de811f7de3126327c4ee267c88bee0ab19d1eb48bb01c3828254"}
Sep 18 13:44:39.687177 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.687159    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kbfb9" event={"ID":"93e9ed3e-0558-49bb-8267-118a66dd4c05","Type":"ContainerStarted","Data":"160cba10ab6985d8e1ec0134dd4dec371590dfcae1f3bc8432a46f766fe041ba"}
Sep 18 13:44:39.689849 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.689833    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:44:39.690208 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.690184    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" event={"ID":"2fbbca64-d3f9-42c6-8175-94b778fbdcb7","Type":"ContainerStarted","Data":"e7a74c8628abee549acbd18aeb7c8fd40684c8c7eea0630b4a4c868cfd1b6862"}
Sep 18 13:44:39.690303 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.690233    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" event={"ID":"2fbbca64-d3f9-42c6-8175-94b778fbdcb7","Type":"ContainerStarted","Data":"10ec5b97e67fcfdf4052a0909811214d57fe3b8a229c9b34be620bada7048e5c"}
Sep 18 13:44:39.690303 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.690262    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" event={"ID":"2fbbca64-d3f9-42c6-8175-94b778fbdcb7","Type":"ContainerStarted","Data":"2559df1d55b9f356163d10d5f08f48c30b20b7e87e3117fd5da90bdc2871dd15"}
Sep 18 13:44:39.690303 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.690304    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" event={"ID":"2fbbca64-d3f9-42c6-8175-94b778fbdcb7","Type":"ContainerStarted","Data":"b4701f11364e5fb41614d4b101ceebb302a88d876e82f741dc090fb29cf60e5a"}
Sep 18 13:44:39.691631 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.691612    2569 generic.go:356] "Generic (PLEG): container finished" podID="f4347c5c0037a0a26f8c1c7c2c1baa4c" containerID="77e5cce35b0e473eba073bd3c81f0839118d0a3cfa1e04844d341795fab1eca8" exitCode=0
Sep 18 13:44:39.691725 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.691686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal" event={"ID":"f4347c5c0037a0a26f8c1c7c2c1baa4c","Type":"ContainerDied","Data":"77e5cce35b0e473eba073bd3c81f0839118d0a3cfa1e04844d341795fab1eca8"}
Sep 18 13:44:39.695792 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.695751    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ww9c2" podStartSLOduration=3.393067684 podStartE2EDuration="21.695734439s" totalImagesPullingTime="18.302666755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:39.695539471 +0000 UTC m=+21.597995656" watchObservedRunningTime="2026-09-18 13:44:39.695734439 +0000 UTC m=+21.598190625"
Sep 18 13:44:39.714191 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.714150    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nvrgd" podStartSLOduration=3.410807564 podStartE2EDuration="21.714134894s" totalImagesPullingTime="18.30332733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:39.713760027 +0000 UTC m=+21.616216223" watchObservedRunningTime="2026-09-18 13:44:39.714134894 +0000 UTC m=+21.616591080"
Sep 18 13:44:39.766491 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:39.766459    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2lqjs" podStartSLOduration=3.450385855 podStartE2EDuration="21.766447586s" totalImagesPullingTime="18.316061731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:39.766332205 +0000 UTC m=+21.668788401" watchObservedRunningTime="2026-09-18 13:44:39.766447586 +0000 UTC m=+21.668903770"
Sep 18 13:44:40.480301 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.480270    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:40.480473 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:40.480428    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:40.494716 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.494636    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T13:44:39.557767692Z","UUID":"0cdd4e4f-41fa-434a-9615-d2636610279f","Handler":null,"Name":"","Endpoint":""}
Sep 18 13:44:40.496504 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.496476    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 13:44:40.496504 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.496508    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 13:44:40.695862 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.695834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal" event={"ID":"f4347c5c0037a0a26f8c1c7c2c1baa4c","Type":"ContainerStarted","Data":"e487223a2b09d40a93cca9aab5ec2d2ee70dbf2a6725752e4bb6c3a37c7be244"}
Sep 18 13:44:40.698410 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.698380    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7" event={"ID":"12b683a1-3f2d-4b34-97e6-d76ed9d00fef","Type":"ContainerStarted","Data":"80b4c6ddf57d046a687c4dfb0ff76c71c135fafd5039c22ea4c2bbe49d09f04c"}
Sep 18 13:44:40.712210 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.712168    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-135.ec2.internal" podStartSLOduration=21.712156827 podStartE2EDuration="21.712156827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:40.711908531 +0000 UTC m=+22.614364715" watchObservedRunningTime="2026-09-18 13:44:40.712156827 +0000 UTC m=+22.614613011"
Sep 18 13:44:40.712336 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.712258    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kbfb9" podStartSLOduration=4.474346141 podStartE2EDuration="22.712248555s" totalImagesPullingTime="18.237902414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:39.781061 +0000 UTC m=+21.683517186" watchObservedRunningTime="2026-09-18 13:44:40.712248555 +0000 UTC m=+22.614704741"
Sep 18 13:44:40.732444 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.732405    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f54l7" podStartSLOduration=3.021375802 podStartE2EDuration="22.732393084s" totalImagesPullingTime="19.711017282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:40.732088367 +0000 UTC m=+22.634544549" watchObservedRunningTime="2026-09-18 13:44:40.732393084 +0000 UTC m=+22.634849271"
Sep 18 13:44:40.755325 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.755298    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:40.755940 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:40.755922    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:41.479549 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:41.479524    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:41.479780 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:41.479627    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:41.479780 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:41.479683    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:41.479780 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:41.479765    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:41.704132 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:41.704101    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:44:41.704615 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:41.704576    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" event={"ID":"2fbbca64-d3f9-42c6-8175-94b778fbdcb7","Type":"ContainerStarted","Data":"7ff62aa5af1b004594b6d0d365e740f580b6c98be628076b5110e80f056f6061"}
Sep 18 13:44:41.784566 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:41.784501    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:41.785184 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:41.785161    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nvrgd"
Sep 18 13:44:42.479774 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:42.479745    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:42.479958 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:42.479883    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:43.479800 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:43.479765    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:43.480230 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:43.479776    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:43.480230 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:43.479864    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:43.480230 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:43.479982    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:43.715968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:43.715623    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:44:43.716636 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:43.716476    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" event={"ID":"2fbbca64-d3f9-42c6-8175-94b778fbdcb7","Type":"ContainerStarted","Data":"91ed2d779b5118cea27eaea6a3c0425894e130469837bb88917e4d45d0f809e8"}
Sep 18 13:44:43.717531 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:43.717025    2569 scope.go:117] "RemoveContainer" containerID="57ef3ffc25d6df249978e3dcbcc84617c8cc30ae556b1b33c1d2b978e3bf695b"
Sep 18 13:44:44.480032 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.479990    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:44.480451 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:44.480124    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:44.721834 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.721813    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:44:44.722216 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.722194    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" event={"ID":"2fbbca64-d3f9-42c6-8175-94b778fbdcb7","Type":"ContainerStarted","Data":"33dd1406205022f26b2c45028b37d71d1b9d54bc5a3c38dd41b7da75e8a1682e"}
Sep 18 13:44:44.722496 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.722472    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:44.722575 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.722543    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:44.722628 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.722590    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:44.724345 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.724319    2569 generic.go:356] "Generic (PLEG): container finished" podID="2406c75d-99cf-4635-a988-6e44c91df885" containerID="ed083c6767410a097bb6b50c8169c6ae956ac7deed2c0361f73e3e778e961b92" exitCode=0
Sep 18 13:44:44.724421 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.724355    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzwlw" event={"ID":"2406c75d-99cf-4635-a988-6e44c91df885","Type":"ContainerDied","Data":"ed083c6767410a097bb6b50c8169c6ae956ac7deed2c0361f73e3e778e961b92"}
Sep 18 13:44:44.737846 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.737791    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:44.738288 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.738270    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:44:44.752025 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:44.751974    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h" podStartSLOduration=8.153889314 podStartE2EDuration="26.751954757s" totalImagesPullingTime="18.598065443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:44.750294782 +0000 UTC m=+26.652751155" watchObservedRunningTime="2026-09-18 13:44:44.751954757 +0000 UTC m=+26.654410943"
Sep 18 13:44:45.445522 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:45.445330    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsq2g"]
Sep 18 13:44:45.445888 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:45.445663    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:45.445888 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:45.445756    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:45.446143 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:45.446127    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fkhh5"]
Sep 18 13:44:45.446231 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:45.446219    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:45.446350 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:45.446312    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:45.446776 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:45.446744    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xrrx8"]
Sep 18 13:44:45.446871 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:45.446840    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:45.446942 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:45.446923    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:45.728138 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:45.728117    2569 generic.go:356] "Generic (PLEG): container finished" podID="2406c75d-99cf-4635-a988-6e44c91df885" containerID="a603159c825c7535c8cdf2fed2633a7bd1987fb77388a4aa741fcd6e7b397547" exitCode=0
Sep 18 13:44:45.728677 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:45.728211    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzwlw" event={"ID":"2406c75d-99cf-4635-a988-6e44c91df885","Type":"ContainerDied","Data":"a603159c825c7535c8cdf2fed2633a7bd1987fb77388a4aa741fcd6e7b397547"}
Sep 18 13:44:46.733178 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:46.733149    2569 generic.go:356] "Generic (PLEG): container finished" podID="2406c75d-99cf-4635-a988-6e44c91df885" containerID="9d8f483fa98bf6a287ea89f865dc47682c55d9ceca763eaa1a41850aa009f792" exitCode=0
Sep 18 13:44:46.733479 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:46.733239    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzwlw" event={"ID":"2406c75d-99cf-4635-a988-6e44c91df885","Type":"ContainerDied","Data":"9d8f483fa98bf6a287ea89f865dc47682c55d9ceca763eaa1a41850aa009f792"}
Sep 18 13:44:47.479661 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:47.479638    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:47.479804 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:47.479638    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:47.479804 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:47.479754    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:47.479896 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:47.479817    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:47.479896 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:47.479638    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:47.479968 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:47.479899    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:49.479585 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:49.479551    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:49.480050 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:49.479559    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:49.480050 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:49.479649    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:49.480050 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:49.479742    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:49.480050 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:49.479569    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:49.480050 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:49.479849    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:51.273129 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:51.273103    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:51.273450 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:51.273136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:51.273450 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.273237    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:51.273450 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.273262    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:51.273450 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.273288    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret podName:4b31645a-f2d0-44ba-aa15-0bc175cb4319 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:23.273272933 +0000 UTC m=+65.175729097 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret") pod "global-pull-secret-syncer-fkhh5" (UID: "4b31645a-f2d0-44ba-aa15-0bc175cb4319") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:44:51.273450 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.273339    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs podName:b2ce9daf-142e-4306-8a94-939272a094d2 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:23.273319125 +0000 UTC m=+65.175775310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs") pod "network-metrics-daemon-xrrx8" (UID: "b2ce9daf-142e-4306-8a94-939272a094d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:44:51.374258 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:51.374208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkjk\" (UniqueName: \"kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk\") pod \"network-check-target-xsq2g\" (UID: \"0105fcdc-d0aa-441c-bb90-83c7419550d6\") " pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:51.374336 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.374307    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:44:51.374336 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.374322    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:44:51.374336 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.374330    2569 projected.go:196] Error preparing data for projected volume kube-api-access-xdkjk for pod openshift-network-diagnostics/network-check-target-xsq2g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:51.374424 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.374374    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk podName:0105fcdc-d0aa-441c-bb90-83c7419550d6 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:23.374360438 +0000 UTC m=+65.276816621 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdkjk" (UniqueName: "kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk") pod "network-check-target-xsq2g" (UID: "0105fcdc-d0aa-441c-bb90-83c7419550d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:44:51.480311 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:51.480289    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:51.480403 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:51.480392    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:51.480440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:51.480402    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:51.480440 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.480400    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq2g" podUID="0105fcdc-d0aa-441c-bb90-83c7419550d6"
Sep 18 13:44:51.480501 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.480467    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrrx8" podUID="b2ce9daf-142e-4306-8a94-939272a094d2"
Sep 18 13:44:51.480549 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:51.480537    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fkhh5" podUID="4b31645a-f2d0-44ba-aa15-0bc175cb4319"
Sep 18 13:44:51.746479 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:51.746449    2569 generic.go:356] "Generic (PLEG): container finished" podID="2406c75d-99cf-4635-a988-6e44c91df885" containerID="398136ad5a83b9c38921d4fbb60722b99fe57acfbeb8a69fab51ede3dcba9226" exitCode=0
Sep 18 13:44:51.746580 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:51.746507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzwlw" event={"ID":"2406c75d-99cf-4635-a988-6e44c91df885","Type":"ContainerDied","Data":"398136ad5a83b9c38921d4fbb60722b99fe57acfbeb8a69fab51ede3dcba9226"}
Sep 18 13:44:52.426835 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.426810    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-135.ec2.internal" event="NodeReady"
Sep 18 13:44:52.427224 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.426918    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 13:44:52.473447 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.473420    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8ksvp"]
Sep 18 13:44:52.489351 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.489324    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.492581 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.492410    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 13:44:52.492581 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.492439    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 13:44:52.495361 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.495336    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8n4vp"]
Sep 18 13:44:52.495517 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.495504    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5gmms\""
Sep 18 13:44:52.513183 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.513162    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8ksvp"]
Sep 18 13:44:52.513271 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.513196    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8n4vp"
Sep 18 13:44:52.513271 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.513203    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8n4vp"]
Sep 18 13:44:52.515752 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.515734    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 13:44:52.515752 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.515751    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 13:44:52.515752 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.515760    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4kzqf\""
Sep 18 13:44:52.515942 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.515760    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 13:44:52.583480 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.583457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d38f5e-d217-4846-a81f-640571391363-tmp-dir\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.583582 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.583493    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfzzt\" (UniqueName: \"kubernetes.io/projected/07d38f5e-d217-4846-a81f-640571391363-kube-api-access-pfzzt\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.583582 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.583526    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.583582 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.583553    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/07d38f5e-d217-4846-a81f-640571391363-config-volume\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.684815 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.684768    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.684815 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.684793    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/07d38f5e-d217-4846-a81f-640571391363-config-volume\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.684815 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.684809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64efa29e-40f6-4f0f-99c5-8fcbb48b687a-cert\") pod \"ingress-canary-8n4vp\" (UID: \"64efa29e-40f6-4f0f-99c5-8fcbb48b687a\") " pod="openshift-ingress-canary/ingress-canary-8n4vp"
Sep 18 13:44:52.684971 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.684840    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jl62\" (UniqueName: \"kubernetes.io/projected/64efa29e-40f6-4f0f-99c5-8fcbb48b687a-kube-api-access-2jl62\") pod \"ingress-canary-8n4vp\" (UID: \"64efa29e-40f6-4f0f-99c5-8fcbb48b687a\") " pod="openshift-ingress-canary/ingress-canary-8n4vp"
Sep 18 13:44:52.684971 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:52.684912    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:44:52.684971 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.684944    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d38f5e-d217-4846-a81f-640571391363-tmp-dir\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.684971 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:52.684971    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls podName:07d38f5e-d217-4846-a81f-640571391363 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:53.184954779 +0000 UTC m=+35.087410960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls") pod "dns-default-8ksvp" (UID: "07d38f5e-d217-4846-a81f-640571391363") : secret "dns-default-metrics-tls" not found
Sep 18 13:44:52.685171 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.684995    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfzzt\" (UniqueName: \"kubernetes.io/projected/07d38f5e-d217-4846-a81f-640571391363-kube-api-access-pfzzt\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.685381 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.685347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d38f5e-d217-4846-a81f-640571391363-tmp-dir\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.685541 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.685526    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/07d38f5e-d217-4846-a81f-640571391363-config-volume\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.698119 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.697950    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfzzt\" (UniqueName: \"kubernetes.io/projected/07d38f5e-d217-4846-a81f-640571391363-kube-api-access-pfzzt\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:52.751234 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.751210    2569 generic.go:356] "Generic (PLEG): container finished" podID="2406c75d-99cf-4635-a988-6e44c91df885" containerID="f58ad58f1e848f8b5fdc2300075f37546ff12b13bf3952b70a761ac8733b0126" exitCode=0
Sep 18 13:44:52.751316 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.751263    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzwlw" event={"ID":"2406c75d-99cf-4635-a988-6e44c91df885","Type":"ContainerDied","Data":"f58ad58f1e848f8b5fdc2300075f37546ff12b13bf3952b70a761ac8733b0126"}
Sep 18 13:44:52.785687 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.785668    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64efa29e-40f6-4f0f-99c5-8fcbb48b687a-cert\") pod \"ingress-canary-8n4vp\" (UID: \"64efa29e-40f6-4f0f-99c5-8fcbb48b687a\") " pod="openshift-ingress-canary/ingress-canary-8n4vp"
Sep 18 13:44:52.785763 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.785713    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jl62\" (UniqueName: \"kubernetes.io/projected/64efa29e-40f6-4f0f-99c5-8fcbb48b687a-kube-api-access-2jl62\") pod \"ingress-canary-8n4vp\" (UID: \"64efa29e-40f6-4f0f-99c5-8fcbb48b687a\") " pod="openshift-ingress-canary/ingress-canary-8n4vp"
Sep 18 13:44:52.787925 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.787910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64efa29e-40f6-4f0f-99c5-8fcbb48b687a-cert\") pod \"ingress-canary-8n4vp\" (UID: \"64efa29e-40f6-4f0f-99c5-8fcbb48b687a\") " pod="openshift-ingress-canary/ingress-canary-8n4vp"
Sep 18 13:44:52.794384 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.794365    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jl62\" (UniqueName: \"kubernetes.io/projected/64efa29e-40f6-4f0f-99c5-8fcbb48b687a-kube-api-access-2jl62\") pod \"ingress-canary-8n4vp\" (UID: \"64efa29e-40f6-4f0f-99c5-8fcbb48b687a\") " pod="openshift-ingress-canary/ingress-canary-8n4vp"
Sep 18 13:44:52.820357 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:52.820339    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8n4vp"
Sep 18 13:44:53.157752 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.157726    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8n4vp"]
Sep 18 13:44:53.162786 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:44:53.162764    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64efa29e_40f6_4f0f_99c5_8fcbb48b687a.slice/crio-43fc6df75f60370d324ca1cb03036eb3efcb25b40cb566bcdd72396de604749e WatchSource:0}: Error finding container 43fc6df75f60370d324ca1cb03036eb3efcb25b40cb566bcdd72396de604749e: Status 404 returned error can't find the container with id 43fc6df75f60370d324ca1cb03036eb3efcb25b40cb566bcdd72396de604749e
Sep 18 13:44:53.188281 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.188254    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:53.188400 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:53.188384    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:44:53.188451 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:53.188436    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls podName:07d38f5e-d217-4846-a81f-640571391363 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:54.188422805 +0000 UTC m=+36.090878994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls") pod "dns-default-8ksvp" (UID: "07d38f5e-d217-4846-a81f-640571391363") : secret "dns-default-metrics-tls" not found
Sep 18 13:44:53.480237 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.480212    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:44:53.480895 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.480344    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:44:53.480895 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.480341    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:44:53.484392 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.484376    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tqxkk\""
Sep 18 13:44:53.484392 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.484382    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 13:44:53.484540 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.484462    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:44:53.484540 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.484468    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 13:44:53.484540 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.484466    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8ktpw\""
Sep 18 13:44:53.484665 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.484470    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:44:53.758283 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.758215    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzwlw" event={"ID":"2406c75d-99cf-4635-a988-6e44c91df885","Type":"ContainerStarted","Data":"df00bf05a1db62193ac89480f00c89f7308a96ec1727b97c3a541e1f28164330"}
Sep 18 13:44:53.759625 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.759601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8n4vp" event={"ID":"64efa29e-40f6-4f0f-99c5-8fcbb48b687a","Type":"ContainerStarted","Data":"43fc6df75f60370d324ca1cb03036eb3efcb25b40cb566bcdd72396de604749e"}
Sep 18 13:44:53.781697 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:53.781653    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jzwlw" podStartSLOduration=7.962487371 podStartE2EDuration="35.781634771s" totalImagesPullingTime="27.8191474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:53.779665355 +0000 UTC m=+35.682121541" watchObservedRunningTime="2026-09-18 13:44:53.781634771 +0000 UTC m=+35.684090962"
Sep 18 13:44:54.195674 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:54.195639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:54.197112 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:54.195842    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:44:54.197112 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:54.195916    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls podName:07d38f5e-d217-4846-a81f-640571391363 nodeName:}" failed. No retries permitted until 2026-09-18 13:44:56.195898472 +0000 UTC m=+38.098354655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls") pod "dns-default-8ksvp" (UID: "07d38f5e-d217-4846-a81f-640571391363") : secret "dns-default-metrics-tls" not found
Sep 18 13:44:56.209448 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:56.209254    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:44:56.209794 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:56.209398    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:44:56.209794 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:44:56.209593    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls podName:07d38f5e-d217-4846-a81f-640571391363 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:00.209572138 +0000 UTC m=+42.112028302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls") pod "dns-default-8ksvp" (UID: "07d38f5e-d217-4846-a81f-640571391363") : secret "dns-default-metrics-tls" not found
Sep 18 13:44:56.766740 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:56.766709    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8n4vp" event={"ID":"64efa29e-40f6-4f0f-99c5-8fcbb48b687a","Type":"ContainerStarted","Data":"5e9013bd22d6c1925b95f63caf823c563c7e91db73194200a3bc7f966d740d0b"}
Sep 18 13:44:56.785581 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:44:56.785533    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8n4vp" podStartSLOduration=2.186167702 podStartE2EDuration="4.785516615s" totalImagesPullingTime="2.599348913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:56.785288822 +0000 UTC m=+38.687745006" watchObservedRunningTime="2026-09-18 13:44:56.785516615 +0000 UTC m=+38.687972800"
Sep 18 13:45:00.237797 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.237762    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:45:00.238174 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:00.237891    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:45:00.238174 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:00.237940    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls podName:07d38f5e-d217-4846-a81f-640571391363 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.237927223 +0000 UTC m=+50.140383406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls") pod "dns-default-8ksvp" (UID: "07d38f5e-d217-4846-a81f-640571391363") : secret "dns-default-metrics-tls" not found
Sep 18 13:45:00.288351 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.288326    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"]
Sep 18 13:45:00.337629 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.337610    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sfzlp"]
Sep 18 13:45:00.337747 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.337729    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.340720 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.340703    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 13:45:00.341015 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.341000    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 13:45:00.341303 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.341283    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 13:45:00.342277 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.342262    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:45:00.342332 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.342262    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sns64\""
Sep 18 13:45:00.354432 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.354410    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-86c7b86bcb-4jjm6"]
Sep 18 13:45:00.354538 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.354516    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.357462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.357446    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nkh8q\""
Sep 18 13:45:00.358431 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.358414    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 13:45:00.358820 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.358801    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 13:45:00.359044 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.358987    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 13:45:00.359752 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.359736    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 13:45:00.364222 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.364204    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 13:45:00.365024 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.365008    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"]
Sep 18 13:45:00.365137 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.365040    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sfzlp"]
Sep 18 13:45:00.365137 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.365062    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86c7b86bcb-4jjm6"]
Sep 18 13:45:00.365137 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.365129    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.367978 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.367961    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 13:45:00.368110 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.367996    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 13:45:00.368214 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.368194    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 13:45:00.368557 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.368538    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 13:45:00.368660 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.368646    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 13:45:00.368710 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.368658    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 13:45:00.368710 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.368647    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p87xc\""
Sep 18 13:45:00.389417 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.389396    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"]
Sep 18 13:45:00.403952 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.403922    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8"]
Sep 18 13:45:00.404053 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.403979    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:00.406897 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.406875    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5l97k\""
Sep 18 13:45:00.407172 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.407148    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 13:45:00.407676 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.407648    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 13:45:00.412214 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.412198    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c8d86cd8-7ctb4"]
Sep 18 13:45:00.412333 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.412317    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8"
Sep 18 13:45:00.414951 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.414934    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pxk4p\""
Sep 18 13:45:00.415036 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.414941    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 13:45:00.415368 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.415355    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:45:00.425058 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.425038    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"]
Sep 18 13:45:00.425170 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.425086    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8"]
Sep 18 13:45:00.425170 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.425109    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c8d86cd8-7ctb4"]
Sep 18 13:45:00.425170 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.425162    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.427781 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.427766    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 13:45:00.428110 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.428093    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 13:45:00.428327 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.428313    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 13:45:00.428436 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.428420    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rp2z9\""
Sep 18 13:45:00.433345 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.433328    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 13:45:00.438751 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.438730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc6c4e02-80c7-463b-8cca-be8f1588caee-tmp\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.438832 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.438759    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p425\" (UniqueName: \"kubernetes.io/projected/cc6c4e02-80c7-463b-8cca-be8f1588caee-kube-api-access-8p425\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.438832 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.438809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cc6c4e02-80c7-463b-8cca-be8f1588caee-snapshots\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.438938 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.438832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6c4e02-80c7-463b-8cca-be8f1588caee-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.438938 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.438871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvtw6\" (UniqueName: \"kubernetes.io/projected/f031d87d-90bc-46f0-b1e1-9cc72ef38e3b-kube-api-access-gvtw6\") pod \"kube-storage-version-migrator-operator-55889f7b47-9rn5l\" (UID: \"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.438938 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.438906    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc6c4e02-80c7-463b-8cca-be8f1588caee-serving-cert\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.439020 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.438935    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6c4e02-80c7-463b-8cca-be8f1588caee-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.439020 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.438977    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f031d87d-90bc-46f0-b1e1-9cc72ef38e3b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-9rn5l\" (UID: \"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.439020 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.439006    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f031d87d-90bc-46f0-b1e1-9cc72ef38e3b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-9rn5l\" (UID: \"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.539978 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.539925    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p425\" (UniqueName: \"kubernetes.io/projected/cc6c4e02-80c7-463b-8cca-be8f1588caee-kube-api-access-8p425\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.539978 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.539954    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6c4e02-80c7-463b-8cca-be8f1588caee-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.539978 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.539971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-installation-pull-secrets\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.540237 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.539986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.540237 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540000    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-certificates\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.540237 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540016    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wtsl\" (UniqueName: \"kubernetes.io/projected/e582da33-0688-485d-90de-91ef855ccc3a-kube-api-access-2wtsl\") pod \"volume-data-source-validator-5c4db8bdf9-4c5m8\" (UID: \"e582da33-0688-485d-90de-91ef855ccc3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8"
Sep 18 13:45:00.540237 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540040    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxfk8\" (UniqueName: \"kubernetes.io/projected/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-kube-api-access-jxfk8\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.540237 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-trusted-ca\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.540237 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540111    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvtw6\" (UniqueName: \"kubernetes.io/projected/f031d87d-90bc-46f0-b1e1-9cc72ef38e3b-kube-api-access-gvtw6\") pod \"kube-storage-version-migrator-operator-55889f7b47-9rn5l\" (UID: \"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.540237 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540127    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.540237 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540185    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-default-certificate\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540237    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-bound-sa-token\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwd2\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-kube-api-access-wcwd2\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540353    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-stats-auth\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/33518b78-9e45-487b-84e7-1a053f46930b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540408    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc6c4e02-80c7-463b-8cca-be8f1588caee-tmp\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540431    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-ca-trust-extracted\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540471    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cc6c4e02-80c7-463b-8cca-be8f1588caee-snapshots\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540501    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc6c4e02-80c7-463b-8cca-be8f1588caee-serving-cert\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540542    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540592    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6c4e02-80c7-463b-8cca-be8f1588caee-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f031d87d-90bc-46f0-b1e1-9cc72ef38e3b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-9rn5l\" (UID: \"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.540679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-image-registry-private-configuration\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.541307 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f031d87d-90bc-46f0-b1e1-9cc72ef38e3b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-9rn5l\" (UID: \"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.541307 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540733    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc6c4e02-80c7-463b-8cca-be8f1588caee-tmp\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.541307 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.540875    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6c4e02-80c7-463b-8cca-be8f1588caee-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.541307 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.541033    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cc6c4e02-80c7-463b-8cca-be8f1588caee-snapshots\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.541307 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.541144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc6c4e02-80c7-463b-8cca-be8f1588caee-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.541486 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.541320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f031d87d-90bc-46f0-b1e1-9cc72ef38e3b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-9rn5l\" (UID: \"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.543534 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.543515    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f031d87d-90bc-46f0-b1e1-9cc72ef38e3b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-9rn5l\" (UID: \"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.543632 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.543561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc6c4e02-80c7-463b-8cca-be8f1588caee-serving-cert\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.549533 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.549513    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvtw6\" (UniqueName: \"kubernetes.io/projected/f031d87d-90bc-46f0-b1e1-9cc72ef38e3b-kube-api-access-gvtw6\") pod \"kube-storage-version-migrator-operator-55889f7b47-9rn5l\" (UID: \"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.550239 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.550219    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p425\" (UniqueName: \"kubernetes.io/projected/cc6c4e02-80c7-463b-8cca-be8f1588caee-kube-api-access-8p425\") pod \"insights-operator-6f9c644dd9-sfzlp\" (UID: \"cc6c4e02-80c7-463b-8cca-be8f1588caee\") " pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.641476 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641428    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-image-registry-private-configuration\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.641476 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-installation-pull-secrets\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.641676 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641481    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.641676 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-certificates\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.641676 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641563    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wtsl\" (UniqueName: \"kubernetes.io/projected/e582da33-0688-485d-90de-91ef855ccc3a-kube-api-access-2wtsl\") pod \"volume-data-source-validator-5c4db8bdf9-4c5m8\" (UID: \"e582da33-0688-485d-90de-91ef855ccc3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8"
Sep 18 13:45:00.641676 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxfk8\" (UniqueName: \"kubernetes.io/projected/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-kube-api-access-jxfk8\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.641676 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:00.641631    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:01.14161244 +0000 UTC m=+43.044068603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:00.641676 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-trusted-ca\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.641968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.641968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641724    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-default-certificate\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.641968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:00.641968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641792    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-bound-sa-token\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.641968 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:00.641804    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:00.641968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641829    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwd2\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-kube-api-access-wcwd2\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.641968 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:00.641870    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:01.141851711 +0000 UTC m=+43.044307885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : secret "router-metrics-certs-default" not found
Sep 18 13:45:00.641968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641894    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-stats-auth\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.641968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641918    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/33518b78-9e45-487b-84e7-1a053f46930b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:00.641968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.641949    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-ca-trust-extracted\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.642456 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.642002    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.642456 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:00.642236    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 13:45:00.642456 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:00.642286    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:00.642456 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:00.642296    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert podName:33518b78-9e45-487b-84e7-1a053f46930b nodeName:}" failed. No retries permitted until 2026-09-18 13:45:01.142279489 +0000 UTC m=+43.044735659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7tmp9" (UID: "33518b78-9e45-487b-84e7-1a053f46930b") : secret "networking-console-plugin-cert" not found
Sep 18 13:45:00.642456 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:00.642299    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8d86cd8-7ctb4: secret "image-registry-tls" not found
Sep 18 13:45:00.642456 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:00.642336    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls podName:a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:01.142325941 +0000 UTC m=+43.044782106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls") pod "image-registry-6c8d86cd8-7ctb4" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54") : secret "image-registry-tls" not found
Sep 18 13:45:00.642456 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.642313    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-certificates\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.642799 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.642634    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-ca-trust-extracted\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.642799 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.642720    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-trusted-ca\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.643277 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.643253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/33518b78-9e45-487b-84e7-1a053f46930b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:00.643865 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.643843    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-installation-pull-secrets\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.643966 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.643949    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-image-registry-private-configuration\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.644195 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.644180    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-default-certificate\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.645002 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.644988    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-stats-auth\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.645855 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.645843    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"
Sep 18 13:45:00.653047 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.653001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwd2\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-kube-api-access-wcwd2\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.653788 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.653764    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxfk8\" (UniqueName: \"kubernetes.io/projected/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-kube-api-access-jxfk8\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:00.654348 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.654327    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wtsl\" (UniqueName: \"kubernetes.io/projected/e582da33-0688-485d-90de-91ef855ccc3a-kube-api-access-2wtsl\") pod \"volume-data-source-validator-5c4db8bdf9-4c5m8\" (UID: \"e582da33-0688-485d-90de-91ef855ccc3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8"
Sep 18 13:45:00.655445 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.655423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-bound-sa-token\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:00.662411 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.662396    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp"
Sep 18 13:45:00.721451 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.721267    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8"
Sep 18 13:45:00.940030 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.940003    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l"]
Sep 18 13:45:00.942899 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:00.942870    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf031d87d_90bc_46f0_b1e1_9cc72ef38e3b.slice/crio-59043df067b50059e18f0ad6c77d8f3e04f009e3a2cff479d1c49dd32f6e3f43 WatchSource:0}: Error finding container 59043df067b50059e18f0ad6c77d8f3e04f009e3a2cff479d1c49dd32f6e3f43: Status 404 returned error can't find the container with id 59043df067b50059e18f0ad6c77d8f3e04f009e3a2cff479d1c49dd32f6e3f43
Sep 18 13:45:00.967143 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:00.967112    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-sfzlp"]
Sep 18 13:45:00.969959 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:00.969937    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6c4e02_80c7_463b_8cca_be8f1588caee.slice/crio-15b64b0203a0c552a09180fa69014e11bc7795ac3d3ff3df3fd9f5383eb5e05e WatchSource:0}: Error finding container 15b64b0203a0c552a09180fa69014e11bc7795ac3d3ff3df3fd9f5383eb5e05e: Status 404 returned error can't find the container with id 15b64b0203a0c552a09180fa69014e11bc7795ac3d3ff3df3fd9f5383eb5e05e
Sep 18 13:45:01.012744 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:01.012684    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8"]
Sep 18 13:45:01.014742 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:01.014724    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode582da33_0688_485d_90de_91ef855ccc3a.slice/crio-e92ffd1cac8f60b23e6d715798f05b107a149949b04ad1381ae91c3689f60fec WatchSource:0}: Error finding container e92ffd1cac8f60b23e6d715798f05b107a149949b04ad1381ae91c3689f60fec: Status 404 returned error can't find the container with id e92ffd1cac8f60b23e6d715798f05b107a149949b04ad1381ae91c3689f60fec
Sep 18 13:45:01.145561 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:01.145514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:01.145561 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:01.145550    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:01.145708 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:01.145571    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:01.145708 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:01.145669    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:02.145654062 +0000 UTC m=+44.048110226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:01.145708 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:01.145688    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:01.145708 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:01.145706    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8d86cd8-7ctb4: secret "image-registry-tls" not found
Sep 18 13:45:01.145906 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:01.145690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:01.145906 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:01.145726    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:01.145906 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:01.145748    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 13:45:01.145906 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:01.145753    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls podName:a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:02.145736225 +0000 UTC m=+44.048192404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls") pod "image-registry-6c8d86cd8-7ctb4" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54") : secret "image-registry-tls" not found
Sep 18 13:45:01.145906 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:01.145785    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert podName:33518b78-9e45-487b-84e7-1a053f46930b nodeName:}" failed. No retries permitted until 2026-09-18 13:45:02.145773597 +0000 UTC m=+44.048229794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7tmp9" (UID: "33518b78-9e45-487b-84e7-1a053f46930b") : secret "networking-console-plugin-cert" not found
Sep 18 13:45:01.145906 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:01.145876    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:02.145864721 +0000 UTC m=+44.048320884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : secret "router-metrics-certs-default" not found
Sep 18 13:45:01.780001 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:01.779966    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l" event={"ID":"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b","Type":"ContainerStarted","Data":"59043df067b50059e18f0ad6c77d8f3e04f009e3a2cff479d1c49dd32f6e3f43"}
Sep 18 13:45:01.781556 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:01.781458    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8" event={"ID":"e582da33-0688-485d-90de-91ef855ccc3a","Type":"ContainerStarted","Data":"e92ffd1cac8f60b23e6d715798f05b107a149949b04ad1381ae91c3689f60fec"}
Sep 18 13:45:01.783379 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:01.783351    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp" event={"ID":"cc6c4e02-80c7-463b-8cca-be8f1588caee","Type":"ContainerStarted","Data":"15b64b0203a0c552a09180fa69014e11bc7795ac3d3ff3df3fd9f5383eb5e05e"}
Sep 18 13:45:02.153912 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:02.153823    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:02.153912 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:02.153885    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:02.154252 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:02.153924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:02.154252 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:02.153963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:02.154252 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:02.154156    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 13:45:02.154252 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:02.154222    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert podName:33518b78-9e45-487b-84e7-1a053f46930b nodeName:}" failed. No retries permitted until 2026-09-18 13:45:04.154202692 +0000 UTC m=+46.056658863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7tmp9" (UID: "33518b78-9e45-487b-84e7-1a053f46930b") : secret "networking-console-plugin-cert" not found
Sep 18 13:45:02.154480 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:02.154294    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:02.154480 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:02.154303    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8d86cd8-7ctb4: secret "image-registry-tls" not found
Sep 18 13:45:02.154480 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:02.154326    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls podName:a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:04.154318744 +0000 UTC m=+46.056774907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls") pod "image-registry-6c8d86cd8-7ctb4" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54") : secret "image-registry-tls" not found
Sep 18 13:45:02.154480 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:02.154379    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:04.15437221 +0000 UTC m=+46.056828373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:02.154480 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:02.154423    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:02.154480 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:02.154441    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:04.154435213 +0000 UTC m=+46.056891376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : secret "router-metrics-certs-default" not found
Sep 18 13:45:04.171394 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:04.171361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:04.171772 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:04.171417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:04.171772 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:04.171523    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:04.171772 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:04.171543    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8d86cd8-7ctb4: secret "image-registry-tls" not found
Sep 18 13:45:04.171772 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:04.171526    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.171508754 +0000 UTC m=+50.073964935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:04.171772 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:04.171608    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:04.171772 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:04.171652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:04.172104 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:04.171781    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 13:45:04.172104 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:04.171829    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert podName:33518b78-9e45-487b-84e7-1a053f46930b nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.171818173 +0000 UTC m=+50.074274336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7tmp9" (UID: "33518b78-9e45-487b-84e7-1a053f46930b") : secret "networking-console-plugin-cert" not found
Sep 18 13:45:04.172104 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:04.171873    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:04.172104 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:04.171875    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls podName:a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.171856711 +0000 UTC m=+50.074312878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls") pod "image-registry-6c8d86cd8-7ctb4" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54") : secret "image-registry-tls" not found
Sep 18 13:45:04.172104 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:04.171896    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.171888731 +0000 UTC m=+50.074344894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : secret "router-metrics-certs-default" not found
Sep 18 13:45:04.792452 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:04.792415    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l" event={"ID":"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b","Type":"ContainerStarted","Data":"b24167b480b2399bb2cf800dd9930c9008b54f5a9c47897f5ce7ceb3fc928c99"}
Sep 18 13:45:04.794128 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:04.794103    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8" event={"ID":"e582da33-0688-485d-90de-91ef855ccc3a","Type":"ContainerStarted","Data":"90d5fc6740b8ae112f7e90cc24be684e762d3c4a69518d3fc81cc677f3b789f4"}
Sep 18 13:45:04.795628 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:04.795609    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp" event={"ID":"cc6c4e02-80c7-463b-8cca-be8f1588caee","Type":"ContainerStarted","Data":"2e1ba94b7f300479dfa6e68ef7cb6af0442b8d516f8f7dab1c8187293bb81f34"}
Sep 18 13:45:04.809775 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:04.809732    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l" podStartSLOduration=1.509192547 podStartE2EDuration="4.809721726s" totalImagesPullingTime="3.300529179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:04.808096413 +0000 UTC m=+46.710552599" watchObservedRunningTime="2026-09-18 13:45:04.809721726 +0000 UTC m=+46.712177910"
Sep 18 13:45:04.823234 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:04.823168    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4c5m8" podStartSLOduration=1.594065515 podStartE2EDuration="4.823153508s" totalImagesPullingTime="3.229087993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:04.822715181 +0000 UTC m=+46.725171367" watchObservedRunningTime="2026-09-18 13:45:04.823153508 +0000 UTC m=+46.725609694"
Sep 18 13:45:04.841514 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:04.841473    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp" podStartSLOduration=1.5621099649999999 podStartE2EDuration="4.8414583s" totalImagesPullingTime="3.279348335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:04.839968016 +0000 UTC m=+46.742424200" watchObservedRunningTime="2026-09-18 13:45:04.8414583 +0000 UTC m=+46.743914487"
Sep 18 13:45:05.247392 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.247365    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5mkl4"]
Sep 18 13:45:05.270086 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.270046    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5mkl4"]
Sep 18 13:45:05.270217 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.270164    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.272942 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.272892    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 13:45:05.273054 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.273021    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 13:45:05.273054 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.273029    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfxkd\""
Sep 18 13:45:05.382993 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.382956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-crio-socket\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.382993 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.382995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jq9z\" (UniqueName: \"kubernetes.io/projected/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-api-access-6jq9z\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.383242 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.383015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.383242 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.383201    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.383339 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.383259    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-data-volume\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.483835 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.483805    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.483982 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.483858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-data-volume\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.483982 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:05.483920    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:05.483982 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:05.483975    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:05.983960073 +0000 UTC m=+47.886416253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:05.484174 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.484048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-crio-socket\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.484174 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.484100    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jq9z\" (UniqueName: \"kubernetes.io/projected/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-api-access-6jq9z\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.484174 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.484129    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.484317 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:05.484220    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:05.484317 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.484231    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-data-volume\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.484317 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:05.484263    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:05.984251812 +0000 UTC m=+47.886707988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:05.484317 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.484264    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-crio-socket\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.493346 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.493326    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jq9z\" (UniqueName: \"kubernetes.io/projected/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-api-access-6jq9z\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.988008 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.987970    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.988182 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:05.988027    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:05.988182 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:05.988089    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:06.988056975 +0000 UTC m=+48.890513160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:05.988182 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:05.988133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:05.988328 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:05.988241    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:05.988328 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:05.988294    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:06.988279077 +0000 UTC m=+48.890735254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:06.996144 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:06.996116    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:06.996507 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:06.996221    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:06.996507 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:06.996233    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:06.996507 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:06.996324    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.996304965 +0000 UTC m=+50.898761148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:06.996507 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:06.996373    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:06.996507 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:06.996417    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:08.996405345 +0000 UTC m=+50.898861524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:07.005499 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:07.005482    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ww9c2_ded613ac-9a58-46d0-9ca6-b66852d872cb/dns-node-resolver/0.log"
Sep 18 13:45:07.607156 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:07.607133    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2lqjs_742dfd2a-3975-49f5-84f5-3bbf7298b399/node-ca/0.log"
Sep 18 13:45:08.055366 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.055340    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-dcq2t"]
Sep 18 13:45:08.070014 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.069997    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.070139 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.069994    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-dcq2t"]
Sep 18 13:45:08.072949 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.072925    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 13:45:08.073053 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.072955    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 13:45:08.073053 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.072955    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 13:45:08.074005 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.073978    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 13:45:08.074057 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.074021    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dgcfm\""
Sep 18 13:45:08.103101 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.103062    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kldwb\" (UniqueName: \"kubernetes.io/projected/ff0ba1e8-8374-4386-a571-acc3e733b18d-kube-api-access-kldwb\") pod \"service-ca-55f6fc7b86-dcq2t\" (UID: \"ff0ba1e8-8374-4386-a571-acc3e733b18d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.103205 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.103181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff0ba1e8-8374-4386-a571-acc3e733b18d-signing-cabundle\") pod \"service-ca-55f6fc7b86-dcq2t\" (UID: \"ff0ba1e8-8374-4386-a571-acc3e733b18d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.103256 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.103214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff0ba1e8-8374-4386-a571-acc3e733b18d-signing-key\") pod \"service-ca-55f6fc7b86-dcq2t\" (UID: \"ff0ba1e8-8374-4386-a571-acc3e733b18d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.203745 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.203721    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff0ba1e8-8374-4386-a571-acc3e733b18d-signing-key\") pod \"service-ca-55f6fc7b86-dcq2t\" (UID: \"ff0ba1e8-8374-4386-a571-acc3e733b18d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.203832 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.203770    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:08.203832 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.203795    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kldwb\" (UniqueName: \"kubernetes.io/projected/ff0ba1e8-8374-4386-a571-acc3e733b18d-kube-api-access-kldwb\") pod \"service-ca-55f6fc7b86-dcq2t\" (UID: \"ff0ba1e8-8374-4386-a571-acc3e733b18d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.203832 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.203821    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:08.203951 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.203839    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:08.203951 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.203861    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:08.203951 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.203898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff0ba1e8-8374-4386-a571-acc3e733b18d-signing-cabundle\") pod \"service-ca-55f6fc7b86-dcq2t\" (UID: \"ff0ba1e8-8374-4386-a571-acc3e733b18d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.204099 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:08.203983    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:45:08.204099 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:08.204002    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 13:45:08.204099 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:08.204007    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8d86cd8-7ctb4: secret "image-registry-tls" not found
Sep 18 13:45:08.204099 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:08.204007    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:08.204099 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:08.204096    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls podName:a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:16.204055974 +0000 UTC m=+58.106512144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls") pod "image-registry-6c8d86cd8-7ctb4" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54") : secret "image-registry-tls" not found
Sep 18 13:45:08.204308 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:08.204125    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:16.204105097 +0000 UTC m=+58.106561274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : secret "router-metrics-certs-default" not found
Sep 18 13:45:08.204308 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:08.204146    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert podName:33518b78-9e45-487b-84e7-1a053f46930b nodeName:}" failed. No retries permitted until 2026-09-18 13:45:16.204136094 +0000 UTC m=+58.106592260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7tmp9" (UID: "33518b78-9e45-487b-84e7-1a053f46930b") : secret "networking-console-plugin-cert" not found
Sep 18 13:45:08.204308 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:08.204161    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:16.204153742 +0000 UTC m=+58.106609905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:08.204487 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.204472    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff0ba1e8-8374-4386-a571-acc3e733b18d-signing-cabundle\") pod \"service-ca-55f6fc7b86-dcq2t\" (UID: \"ff0ba1e8-8374-4386-a571-acc3e733b18d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.206224 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.206208    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff0ba1e8-8374-4386-a571-acc3e733b18d-signing-key\") pod \"service-ca-55f6fc7b86-dcq2t\" (UID: \"ff0ba1e8-8374-4386-a571-acc3e733b18d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.212574 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.212555    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kldwb\" (UniqueName: \"kubernetes.io/projected/ff0ba1e8-8374-4386-a571-acc3e733b18d-kube-api-access-kldwb\") pod \"service-ca-55f6fc7b86-dcq2t\" (UID: \"ff0ba1e8-8374-4386-a571-acc3e733b18d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.304667 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.304647    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:45:08.304735 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:08.304727    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:45:08.304784 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:08.304775    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls podName:07d38f5e-d217-4846-a81f-640571391363 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:24.304762366 +0000 UTC m=+66.207218546 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls") pod "dns-default-8ksvp" (UID: "07d38f5e-d217-4846-a81f-640571391363") : secret "dns-default-metrics-tls" not found
Sep 18 13:45:08.381498 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.381456    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t"
Sep 18 13:45:08.405502 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.405484    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8n4vp_64efa29e-40f6-4f0f-99c5-8fcbb48b687a/serve-healthcheck-canary/0.log"
Sep 18 13:45:08.674793 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.674731    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-dcq2t"]
Sep 18 13:45:08.677918 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:08.677897    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0ba1e8_8374_4386_a571_acc3e733b18d.slice/crio-e0d0f8a3bd0d7d96171bb46e8aa99409b0370a7a31e8529f40405d0f65df21bc WatchSource:0}: Error finding container e0d0f8a3bd0d7d96171bb46e8aa99409b0370a7a31e8529f40405d0f65df21bc: Status 404 returned error can't find the container with id e0d0f8a3bd0d7d96171bb46e8aa99409b0370a7a31e8529f40405d0f65df21bc
Sep 18 13:45:08.806339 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:08.806315    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t" event={"ID":"ff0ba1e8-8374-4386-a571-acc3e733b18d","Type":"ContainerStarted","Data":"e0d0f8a3bd0d7d96171bb46e8aa99409b0370a7a31e8529f40405d0f65df21bc"}
Sep 18 13:45:09.009142 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:09.009121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:09.009242 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:09.009209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:09.009287 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:09.009233    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:09.009287 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:09.009286    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:13.009272535 +0000 UTC m=+54.911728698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:09.009371 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:09.009357    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:09.009413 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:09.009405    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:13.009393878 +0000 UTC m=+54.911850041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:10.813690 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:10.813613    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t" event={"ID":"ff0ba1e8-8374-4386-a571-acc3e733b18d","Type":"ContainerStarted","Data":"2418c1149649640456eac1c4c5a333b4edc5a62425f47f379cbb3cd1af0287bd"}
Sep 18 13:45:10.830407 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:10.830362    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-dcq2t" podStartSLOduration=0.946104251 podStartE2EDuration="2.83034714s" totalImagesPullingTime="1.884242889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:10.829876076 +0000 UTC m=+52.732332261" watchObservedRunningTime="2026-09-18 13:45:10.83034714 +0000 UTC m=+52.732803325"
Sep 18 13:45:13.042963 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:13.042929    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:13.043492 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:13.043004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:13.043492 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:13.043083    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:13.043492 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:13.043170    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:21.043154997 +0000 UTC m=+62.945611174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:13.043492 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:13.043087    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:45:13.043492 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:13.043217    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:21.043204264 +0000 UTC m=+62.945660441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : configmap "kube-rbac-proxy" not found
Sep 18 13:45:16.270101 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.270059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:16.270479 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.270123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:16.270479 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.270142    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:16.270479 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.270164    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:16.270479 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:16.270260    2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 13:45:16.270479 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:16.270274    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:45:16.270479 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:16.270308    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:32.270288353 +0000 UTC m=+74.172744538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : configmap references non-existent config key: service-ca.crt
Sep 18 13:45:16.270479 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:16.270327    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert podName:33518b78-9e45-487b-84e7-1a053f46930b nodeName:}" failed. No retries permitted until 2026-09-18 13:45:32.270318452 +0000 UTC m=+74.172774619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-7tmp9" (UID: "33518b78-9e45-487b-84e7-1a053f46930b") : secret "networking-console-plugin-cert" not found
Sep 18 13:45:16.270479 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:16.270340    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs podName:9ddf2987-06c4-4c7e-a2a8-f89293cc0497 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:32.270332441 +0000 UTC m=+74.172788606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs") pod "router-default-86c7b86bcb-4jjm6" (UID: "9ddf2987-06c4-4c7e-a2a8-f89293cc0497") : secret "router-metrics-certs-default" not found
Sep 18 13:45:16.272460 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.272440    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls\") pod \"image-registry-6c8d86cd8-7ctb4\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") " pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:16.333320 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.333303    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:16.605462 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.605433    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c8d86cd8-7ctb4"]
Sep 18 13:45:16.606801 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:16.606779    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ac64cd_1f2f_4cd2_b651_4ef63ef14c54.slice/crio-1a40b797f9a2e11423ff0e4b6e629389db86786f59921b33a4b956da014efa22 WatchSource:0}: Error finding container 1a40b797f9a2e11423ff0e4b6e629389db86786f59921b33a4b956da014efa22: Status 404 returned error can't find the container with id 1a40b797f9a2e11423ff0e4b6e629389db86786f59921b33a4b956da014efa22
Sep 18 13:45:16.743380 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.743354    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wcb2h"
Sep 18 13:45:16.835605 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.835566    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4" event={"ID":"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54","Type":"ContainerStarted","Data":"4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8"}
Sep 18 13:45:16.835769 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.835648    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4" event={"ID":"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54","Type":"ContainerStarted","Data":"1a40b797f9a2e11423ff0e4b6e629389db86786f59921b33a4b956da014efa22"}
Sep 18 13:45:16.835769 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.835692    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:16.858583 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:16.858490    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4" podStartSLOduration=16.858473320999998 podStartE2EDuration="16.858473321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:16.858184238 +0000 UTC m=+58.760640425" watchObservedRunningTime="2026-09-18 13:45:16.858473321 +0000 UTC m=+58.760929507"
Sep 18 13:45:21.109024 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:21.108992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:21.109469 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:21.109061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:21.109469 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:21.109174    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:21.109469 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:21.109245    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls podName:d4d7f5a1-3d5c-446d-b9df-8cf78eed185a nodeName:}" failed. No retries permitted until 2026-09-18 13:45:37.109225231 +0000 UTC m=+79.011681409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5mkl4" (UID: "d4d7f5a1-3d5c-446d-b9df-8cf78eed185a") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:45:21.109619 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:21.109507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:23.326333 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.326304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:45:23.326670 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.326340    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:45:23.329483 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.329467    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:45:23.330652 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.330638    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:45:23.339310 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.339286    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b31645a-f2d0-44ba-aa15-0bc175cb4319-original-pull-secret\") pod \"global-pull-secret-syncer-fkhh5\" (UID: \"4b31645a-f2d0-44ba-aa15-0bc175cb4319\") " pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:45:23.339417 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.339317    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ce9daf-142e-4306-8a94-939272a094d2-metrics-certs\") pod \"network-metrics-daemon-xrrx8\" (UID: \"b2ce9daf-142e-4306-8a94-939272a094d2\") " pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:45:23.427525 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.427501    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkjk\" (UniqueName: \"kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk\") pod \"network-check-target-xsq2g\" (UID: \"0105fcdc-d0aa-441c-bb90-83c7419550d6\") " pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:45:23.430417 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.430402    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 13:45:23.440979 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.440966    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 13:45:23.450488 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.450467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdkjk\" (UniqueName: \"kubernetes.io/projected/0105fcdc-d0aa-441c-bb90-83c7419550d6-kube-api-access-xdkjk\") pod \"network-check-target-xsq2g\" (UID: \"0105fcdc-d0aa-441c-bb90-83c7419550d6\") " pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:45:23.490567 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.490546    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fkhh5"
Sep 18 13:45:23.497496 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.497478    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tqxkk\""
Sep 18 13:45:23.502326 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.502305    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8ktpw\""
Sep 18 13:45:23.504566 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.504548    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrrx8"
Sep 18 13:45:23.510494 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.510479    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:45:23.761814 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.761792    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fkhh5"]
Sep 18 13:45:23.763855 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:23.763829    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b31645a_f2d0_44ba_aa15_0bc175cb4319.slice/crio-9def8edb187e6dc084769d1cf7b2cac1fd1b2d068a7b41d6d06b408375d2bdc5 WatchSource:0}: Error finding container 9def8edb187e6dc084769d1cf7b2cac1fd1b2d068a7b41d6d06b408375d2bdc5: Status 404 returned error can't find the container with id 9def8edb187e6dc084769d1cf7b2cac1fd1b2d068a7b41d6d06b408375d2bdc5
Sep 18 13:45:23.787404 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.787387    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xrrx8"]
Sep 18 13:45:23.788362 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.788345    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsq2g"]
Sep 18 13:45:23.789138 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:23.789113    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ce9daf_142e_4306_8a94_939272a094d2.slice/crio-ba631f0214e300f1d74f08398e49a3984f4ba53fe400821ffec82c4b40cdcde3 WatchSource:0}: Error finding container ba631f0214e300f1d74f08398e49a3984f4ba53fe400821ffec82c4b40cdcde3: Status 404 returned error can't find the container with id ba631f0214e300f1d74f08398e49a3984f4ba53fe400821ffec82c4b40cdcde3
Sep 18 13:45:23.790346 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:23.790323    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0105fcdc_d0aa_441c_bb90_83c7419550d6.slice/crio-63a2261018fb8b8865ca5127dea8d96c362ab5e03fffcf2f75e257df830adbb9 WatchSource:0}: Error finding container 63a2261018fb8b8865ca5127dea8d96c362ab5e03fffcf2f75e257df830adbb9: Status 404 returned error can't find the container with id 63a2261018fb8b8865ca5127dea8d96c362ab5e03fffcf2f75e257df830adbb9
Sep 18 13:45:23.856040 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.855989    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xrrx8" event={"ID":"b2ce9daf-142e-4306-8a94-939272a094d2","Type":"ContainerStarted","Data":"ba631f0214e300f1d74f08398e49a3984f4ba53fe400821ffec82c4b40cdcde3"}
Sep 18 13:45:23.857078 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.857051    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsq2g" event={"ID":"0105fcdc-d0aa-441c-bb90-83c7419550d6","Type":"ContainerStarted","Data":"63a2261018fb8b8865ca5127dea8d96c362ab5e03fffcf2f75e257df830adbb9"}
Sep 18 13:45:23.858113 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:23.858093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fkhh5" event={"ID":"4b31645a-f2d0-44ba-aa15-0bc175cb4319","Type":"ContainerStarted","Data":"9def8edb187e6dc084769d1cf7b2cac1fd1b2d068a7b41d6d06b408375d2bdc5"}
Sep 18 13:45:24.334366 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:24.334251    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:45:24.337298 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:24.337265    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07d38f5e-d217-4846-a81f-640571391363-metrics-tls\") pod \"dns-default-8ksvp\" (UID: \"07d38f5e-d217-4846-a81f-640571391363\") " pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:45:24.604833 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:24.604760    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5gmms\""
Sep 18 13:45:24.612502 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:24.612470    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:45:24.920325 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:24.920292    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8ksvp"]
Sep 18 13:45:24.983301 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:24.983260    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d38f5e_d217_4846_a81f_640571391363.slice/crio-32dc63d40dab38eb7e3a1a3675e10d82f69e8244130673891be12b1110b95f6a WatchSource:0}: Error finding container 32dc63d40dab38eb7e3a1a3675e10d82f69e8244130673891be12b1110b95f6a: Status 404 returned error can't find the container with id 32dc63d40dab38eb7e3a1a3675e10d82f69e8244130673891be12b1110b95f6a
Sep 18 13:45:25.868842 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:25.868806    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8ksvp" event={"ID":"07d38f5e-d217-4846-a81f-640571391363","Type":"ContainerStarted","Data":"32dc63d40dab38eb7e3a1a3675e10d82f69e8244130673891be12b1110b95f6a"}
Sep 18 13:45:25.871318 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:25.871287    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xrrx8" event={"ID":"b2ce9daf-142e-4306-8a94-939272a094d2","Type":"ContainerStarted","Data":"3767a12ff183e29807f9b9369efac018251e3a932c567573984effcbbe1303d1"}
Sep 18 13:45:25.871448 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:25.871344    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xrrx8" event={"ID":"b2ce9daf-142e-4306-8a94-939272a094d2","Type":"ContainerStarted","Data":"4fb0232f0999d9fcfb04a038b737b9f66f70d687fbd46fc6598c0548697749e2"}
Sep 18 13:45:25.895749 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:25.895624    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xrrx8" podStartSLOduration=66.677365597 podStartE2EDuration="1m7.895610008s" totalImagesPullingTime="1.218244411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:25.894196565 +0000 UTC m=+67.796652773" watchObservedRunningTime="2026-09-18 13:45:25.895610008 +0000 UTC m=+67.798066210"
Sep 18 13:45:28.889230 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:28.889195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8ksvp" event={"ID":"07d38f5e-d217-4846-a81f-640571391363","Type":"ContainerStarted","Data":"36ae3fdb54721c7a9d1752f8dcd0209c0a20aaeb37f9a134959b6dce082342a0"}
Sep 18 13:45:28.891005 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:28.890978    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsq2g" event={"ID":"0105fcdc-d0aa-441c-bb90-83c7419550d6","Type":"ContainerStarted","Data":"80a47524bba05360421df88da49bca5a7e01256ca293b9bc3ebafaa6b512595e"}
Sep 18 13:45:28.891121 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:28.891057    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:45:28.912248 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:28.912198    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xsq2g" podStartSLOduration=65.946493474 podStartE2EDuration="1m10.91217819s" totalImagesPullingTime="4.965684716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:28.910419638 +0000 UTC m=+70.812875824" watchObservedRunningTime="2026-09-18 13:45:28.91217819 +0000 UTC m=+70.814634376"
Sep 18 13:45:29.545750 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.545717    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"]
Sep 18 13:45:29.547863 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.547849    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.550921 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.550886    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 13:45:29.551334 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.551317    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 13:45:29.552527 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.552508    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 13:45:29.552629 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.552556    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 13:45:29.558584 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.558563    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"]
Sep 18 13:45:29.560437 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.560424    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.563349 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.563329    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 13:45:29.563430 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.563330    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 13:45:29.563588 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.563575    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 13:45:29.563658 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.563644    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 13:45:29.569646 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.569629    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"]
Sep 18 13:45:29.586846 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.586826    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"]
Sep 18 13:45:29.630250 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.630232    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8d86cd8-7ctb4"]
Sep 18 13:45:29.676793 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.676770    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7596bf68dd-lr7t4"]
Sep 18 13:45:29.678683 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.678669    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.679057 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.679038    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-ca\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.679125 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.679084    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9dn\" (UniqueName: \"kubernetes.io/projected/114b13c3-400f-47db-8ae1-d396201c8f76-kube-api-access-jg9dn\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.679125 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.679109    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.679205 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.679166    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e79f88-7c30-4d2c-a363-0712670912cf-tmp\") pod \"klusterlet-addon-workmgr-d49db8546-nj4m8\" (UID: \"65e79f88-7c30-4d2c-a363-0712670912cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.679205 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.679193    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.679262 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.679209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l99ff\" (UniqueName: \"kubernetes.io/projected/65e79f88-7c30-4d2c-a363-0712670912cf-kube-api-access-l99ff\") pod \"klusterlet-addon-workmgr-d49db8546-nj4m8\" (UID: \"65e79f88-7c30-4d2c-a363-0712670912cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.679303 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.679265    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-hub\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.679303 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.679280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/114b13c3-400f-47db-8ae1-d396201c8f76-additional-service-ca\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.679389 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.679315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/65e79f88-7c30-4d2c-a363-0712670912cf-klusterlet-config\") pod \"klusterlet-addon-workmgr-d49db8546-nj4m8\" (UID: \"65e79f88-7c30-4d2c-a363-0712670912cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.698589 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.698567    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7596bf68dd-lr7t4"]
Sep 18 13:45:29.779554 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.779654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779557    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l99ff\" (UniqueName: \"kubernetes.io/projected/65e79f88-7c30-4d2c-a363-0712670912cf-kube-api-access-l99ff\") pod \"klusterlet-addon-workmgr-d49db8546-nj4m8\" (UID: \"65e79f88-7c30-4d2c-a363-0712670912cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.779654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779575    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-registry-tls\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.779654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779592    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-installation-pull-secrets\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.779654 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-bound-sa-token\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.779793 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779655    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg2wf\" (UniqueName: \"kubernetes.io/projected/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-kube-api-access-xg2wf\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.779793 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-hub\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.779793 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779719    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/114b13c3-400f-47db-8ae1-d396201c8f76-additional-service-ca\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.779793 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779771    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/65e79f88-7c30-4d2c-a363-0712670912cf-klusterlet-config\") pod \"klusterlet-addon-workmgr-d49db8546-nj4m8\" (UID: \"65e79f88-7c30-4d2c-a363-0712670912cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.779929 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779801    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-ca\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.779929 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779825    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9dn\" (UniqueName: \"kubernetes.io/projected/114b13c3-400f-47db-8ae1-d396201c8f76-kube-api-access-jg9dn\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.779929 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779851    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.779929 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-image-registry-private-configuration\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.779929 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779907    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-ca-trust-extracted\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.780192 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-registry-certificates\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.780192 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.779972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e79f88-7c30-4d2c-a363-0712670912cf-tmp\") pod \"klusterlet-addon-workmgr-d49db8546-nj4m8\" (UID: \"65e79f88-7c30-4d2c-a363-0712670912cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.780330 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.780313    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65e79f88-7c30-4d2c-a363-0712670912cf-tmp\") pod \"klusterlet-addon-workmgr-d49db8546-nj4m8\" (UID: \"65e79f88-7c30-4d2c-a363-0712670912cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.780559 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.780326    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-trusted-ca\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.780652 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.780632    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/114b13c3-400f-47db-8ae1-d396201c8f76-additional-service-ca\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.782296 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.782277    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.782296 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.782286    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-ca\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.782660 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.782642    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/65e79f88-7c30-4d2c-a363-0712670912cf-klusterlet-config\") pod \"klusterlet-addon-workmgr-d49db8546-nj4m8\" (UID: \"65e79f88-7c30-4d2c-a363-0712670912cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.782731 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.782716    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-hub\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.782814 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.782798    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/114b13c3-400f-47db-8ae1-d396201c8f76-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.805816 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.805771    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l99ff\" (UniqueName: \"kubernetes.io/projected/65e79f88-7c30-4d2c-a363-0712670912cf-kube-api-access-l99ff\") pod \"klusterlet-addon-workmgr-d49db8546-nj4m8\" (UID: \"65e79f88-7c30-4d2c-a363-0712670912cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.807213 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.807195    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9dn\" (UniqueName: \"kubernetes.io/projected/114b13c3-400f-47db-8ae1-d396201c8f76-kube-api-access-jg9dn\") pod \"cluster-proxy-proxy-agent-98b99d59f-ggr8q\" (UID: \"114b13c3-400f-47db-8ae1-d396201c8f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.856537 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.856513    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:29.876994 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.876975    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"
Sep 18 13:45:29.881227 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.881205    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-image-registry-private-configuration\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.881302 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.881233    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-ca-trust-extracted\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.881302 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.881255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-registry-certificates\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.881302 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.881284    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-trusted-ca\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.881470 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.881313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-registry-tls\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.881470 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.881337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-installation-pull-secrets\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.881470 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.881361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-bound-sa-token\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.881470 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.881389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg2wf\" (UniqueName: \"kubernetes.io/projected/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-kube-api-access-xg2wf\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.881808 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.881766    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-ca-trust-extracted\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.882201 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.882169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-registry-certificates\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.882573 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.882550    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-trusted-ca\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.884291 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.884266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-image-registry-private-configuration\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.884568 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.884545    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-installation-pull-secrets\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.884568 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.884559    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-registry-tls\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.899171 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.899091    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8ksvp" event={"ID":"07d38f5e-d217-4846-a81f-640571391363","Type":"ContainerStarted","Data":"57253dd658710e71061aa92fd6142d86d3f666638eee99726aa039d860bd3ccf"}
Sep 18 13:45:29.899517 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.899295    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:45:29.901098 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.901050    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fkhh5" event={"ID":"4b31645a-f2d0-44ba-aa15-0bc175cb4319","Type":"ContainerStarted","Data":"f4d6372cad25129ab61a6fc18dc3314a3151cd9812a3685329cf6483b844f291"}
Sep 18 13:45:29.904021 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.903972    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-bound-sa-token\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.904163 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.904147    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg2wf\" (UniqueName: \"kubernetes.io/projected/48c4cebb-4fe4-42c8-9bb0-0336a139bfdb-kube-api-access-xg2wf\") pod \"image-registry-7596bf68dd-lr7t4\" (UID: \"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb\") " pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:29.938185 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.938090    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8ksvp" podStartSLOduration=34.164249284 podStartE2EDuration="37.938051456s" totalImagesPullingTime="3.773802172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:29.93413304 +0000 UTC m=+71.836589226" watchObservedRunningTime="2026-09-18 13:45:29.938051456 +0000 UTC m=+71.840507643"
Sep 18 13:45:29.958552 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.958512    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fkhh5" podStartSLOduration=66.950252182 podStartE2EDuration="1m11.958499129s" totalImagesPullingTime="5.008246947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:29.957707093 +0000 UTC m=+71.860163289" watchObservedRunningTime="2026-09-18 13:45:29.958499129 +0000 UTC m=+71.860955311"
Sep 18 13:45:29.986038 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:29.986021    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:30.143488 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:30.143421    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"]
Sep 18 13:45:30.147490 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:30.147465    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e79f88_7c30_4d2c_a363_0712670912cf.slice/crio-15d1a58ed84208242c81334816c65b2d31621494eecae92a2783c99dc5e95285 WatchSource:0}: Error finding container 15d1a58ed84208242c81334816c65b2d31621494eecae92a2783c99dc5e95285: Status 404 returned error can't find the container with id 15d1a58ed84208242c81334816c65b2d31621494eecae92a2783c99dc5e95285
Sep 18 13:45:30.165386 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:30.165363    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q"]
Sep 18 13:45:30.167450 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:30.167428    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod114b13c3_400f_47db_8ae1_d396201c8f76.slice/crio-04e73ee43b04897ef7e7bc6d99ab82289852479cd20d18f1e3f88698d21bcfbc WatchSource:0}: Error finding container 04e73ee43b04897ef7e7bc6d99ab82289852479cd20d18f1e3f88698d21bcfbc: Status 404 returned error can't find the container with id 04e73ee43b04897ef7e7bc6d99ab82289852479cd20d18f1e3f88698d21bcfbc
Sep 18 13:45:30.254416 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:30.254394    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7596bf68dd-lr7t4"]
Sep 18 13:45:30.260120 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:30.260098    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c4cebb_4fe4_42c8_9bb0_0336a139bfdb.slice/crio-0cbde65293acda66e0ad467387b5ae160fe3b7e45defd9d1ff8b8be84cfba7e8 WatchSource:0}: Error finding container 0cbde65293acda66e0ad467387b5ae160fe3b7e45defd9d1ff8b8be84cfba7e8: Status 404 returned error can't find the container with id 0cbde65293acda66e0ad467387b5ae160fe3b7e45defd9d1ff8b8be84cfba7e8
Sep 18 13:45:30.910113 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:30.909790    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4" event={"ID":"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb","Type":"ContainerStarted","Data":"937d877219509bc047d0cf70f51cfeeba5db68bfadbddfcb1a444d3a74204ebd"}
Sep 18 13:45:30.910113 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:30.909849    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4" event={"ID":"48c4cebb-4fe4-42c8-9bb0-0336a139bfdb","Type":"ContainerStarted","Data":"0cbde65293acda66e0ad467387b5ae160fe3b7e45defd9d1ff8b8be84cfba7e8"}
Sep 18 13:45:30.910113 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:30.909917    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:30.911324 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:30.911293    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8" event={"ID":"65e79f88-7c30-4d2c-a363-0712670912cf","Type":"ContainerStarted","Data":"15d1a58ed84208242c81334816c65b2d31621494eecae92a2783c99dc5e95285"}
Sep 18 13:45:30.912830 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:30.912805    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q" event={"ID":"114b13c3-400f-47db-8ae1-d396201c8f76","Type":"ContainerStarted","Data":"04e73ee43b04897ef7e7bc6d99ab82289852479cd20d18f1e3f88698d21bcfbc"}
Sep 18 13:45:30.933391 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:30.933352    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4" podStartSLOduration=1.933339275 podStartE2EDuration="1.933339275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:30.932580552 +0000 UTC m=+72.835036763" watchObservedRunningTime="2026-09-18 13:45:30.933339275 +0000 UTC m=+72.835795456"
Sep 18 13:45:32.303468 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:32.303432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:32.303921 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:32.303490    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:32.303921 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:32.303518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:32.304707 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:32.304676    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-service-ca-bundle\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:32.309867 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:32.309823    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddf2987-06c4-4c7e-a2a8-f89293cc0497-metrics-certs\") pod \"router-default-86c7b86bcb-4jjm6\" (UID: \"9ddf2987-06c4-4c7e-a2a8-f89293cc0497\") " pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:32.310938 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:32.310914    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33518b78-9e45-487b-84e7-1a053f46930b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7tmp9\" (UID: \"33518b78-9e45-487b-84e7-1a053f46930b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:32.475514 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:32.475459    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p87xc\""
Sep 18 13:45:32.482842 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:32.482643    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:32.517211 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:32.517005    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5l97k\""
Sep 18 13:45:32.524529 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:32.524501    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"
Sep 18 13:45:36.241349 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.241326    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9"]
Sep 18 13:45:36.242881 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:36.242859    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33518b78_9e45_487b_84e7_1a053f46930b.slice/crio-5f3199143066ea84860c5a609bfa15fe559b351355505bf90223d1fca46560b6 WatchSource:0}: Error finding container 5f3199143066ea84860c5a609bfa15fe559b351355505bf90223d1fca46560b6: Status 404 returned error can't find the container with id 5f3199143066ea84860c5a609bfa15fe559b351355505bf90223d1fca46560b6
Sep 18 13:45:36.256045 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.256026    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86c7b86bcb-4jjm6"]
Sep 18 13:45:36.258050 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:36.258030    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ddf2987_06c4_4c7e_a2a8_f89293cc0497.slice/crio-c060513fd978c232b75ba938cab168fc68d076b2a2dab29becbb1fa515c0d5be WatchSource:0}: Error finding container c060513fd978c232b75ba938cab168fc68d076b2a2dab29becbb1fa515c0d5be: Status 404 returned error can't find the container with id c060513fd978c232b75ba938cab168fc68d076b2a2dab29becbb1fa515c0d5be
Sep 18 13:45:36.941417 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.941320    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q" event={"ID":"114b13c3-400f-47db-8ae1-d396201c8f76","Type":"ContainerStarted","Data":"47de1b784ff92ba9a16107936335efb2c17aa357aa9b701d3c9f62b1e7177d41"}
Sep 18 13:45:36.941417 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.941374    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q" event={"ID":"114b13c3-400f-47db-8ae1-d396201c8f76","Type":"ContainerStarted","Data":"b4895330ef3a5a4d699d4dad32d75e0ab6d1d811d21d27d175d98229284eff2a"}
Sep 18 13:45:36.941417 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.941393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q" event={"ID":"114b13c3-400f-47db-8ae1-d396201c8f76","Type":"ContainerStarted","Data":"4b5c1a47715b61d1997dd6d54049298f405a66482306a19946801334a66f3795"}
Sep 18 13:45:36.943346 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.943316    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86c7b86bcb-4jjm6" event={"ID":"9ddf2987-06c4-4c7e-a2a8-f89293cc0497","Type":"ContainerStarted","Data":"a217263040037a36163d2894216740c11ea57cd538c676d04824f3d3f4e72970"}
Sep 18 13:45:36.943499 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.943373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86c7b86bcb-4jjm6" event={"ID":"9ddf2987-06c4-4c7e-a2a8-f89293cc0497","Type":"ContainerStarted","Data":"c060513fd978c232b75ba938cab168fc68d076b2a2dab29becbb1fa515c0d5be"}
Sep 18 13:45:36.944983 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.944950    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8" event={"ID":"65e79f88-7c30-4d2c-a363-0712670912cf","Type":"ContainerStarted","Data":"653234f888e6b05155958964a8ea62bdf5a358f0db7bb1985228d506db988d6c"}
Sep 18 13:45:36.945174 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.945105    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:36.946238 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.946216    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9" event={"ID":"33518b78-9e45-487b-84e7-1a053f46930b","Type":"ContainerStarted","Data":"5f3199143066ea84860c5a609bfa15fe559b351355505bf90223d1fca46560b6"}
Sep 18 13:45:36.946921 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.946902    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8"
Sep 18 13:45:36.962402 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.962361    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-98b99d59f-ggr8q" podStartSLOduration=2.160838497 podStartE2EDuration="7.962342565s" totalImagesPullingTime="5.801504068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:36.961345669 +0000 UTC m=+78.863801855" watchObservedRunningTime="2026-09-18 13:45:36.962342565 +0000 UTC m=+78.864798732"
Sep 18 13:45:36.979234 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:36.979090    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d49db8546-nj4m8" podStartSLOduration=2.136706049 podStartE2EDuration="7.97905938s" totalImagesPullingTime="5.842353331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:36.977927795 +0000 UTC m=+78.880383979" watchObservedRunningTime="2026-09-18 13:45:36.97905938 +0000 UTC m=+78.881515599"
Sep 18 13:45:37.000140 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.000098    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-86c7b86bcb-4jjm6" podStartSLOduration=37.000044074 podStartE2EDuration="37.000044074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:36.998919616 +0000 UTC m=+78.901375804" watchObservedRunningTime="2026-09-18 13:45:37.000044074 +0000 UTC m=+78.902500282"
Sep 18 13:45:37.145595 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.145561    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:37.148298 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.148269    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4d7f5a1-3d5c-446d-b9df-8cf78eed185a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5mkl4\" (UID: \"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a\") " pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:37.381038 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.381015    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfxkd\""
Sep 18 13:45:37.389285 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.389270    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5mkl4"
Sep 18 13:45:37.483235 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.483213    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:37.485527 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.485508    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:37.655379 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.655355    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5mkl4"]
Sep 18 13:45:37.658147 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:37.658121    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d7f5a1_3d5c_446d_b9df_8cf78eed185a.slice/crio-ac75bfa6c8b9bb1e7ff30cc015d0d329ce2c55083debffab5eb5e4009060a86f WatchSource:0}: Error finding container ac75bfa6c8b9bb1e7ff30cc015d0d329ce2c55083debffab5eb5e4009060a86f: Status 404 returned error can't find the container with id ac75bfa6c8b9bb1e7ff30cc015d0d329ce2c55083debffab5eb5e4009060a86f
Sep 18 13:45:37.951774 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.951740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9" event={"ID":"33518b78-9e45-487b-84e7-1a053f46930b","Type":"ContainerStarted","Data":"5d5c77542a475186492ac91fda63955ee6c2354d65698cacffc8956bcf17c390"}
Sep 18 13:45:37.953465 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.953440    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5mkl4" event={"ID":"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a","Type":"ContainerStarted","Data":"dd4cf83ddfe7e9934e0cf794155a1e4e607252f40986a470b1bd371e87cf0950"}
Sep 18 13:45:37.953550 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.953486    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5mkl4" event={"ID":"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a","Type":"ContainerStarted","Data":"ac75bfa6c8b9bb1e7ff30cc015d0d329ce2c55083debffab5eb5e4009060a86f"}
Sep 18 13:45:37.953926 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.953899    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:37.954834 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.954812    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-86c7b86bcb-4jjm6"
Sep 18 13:45:37.968085 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:37.968035    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7tmp9" podStartSLOduration=37.018654449 podStartE2EDuration="37.968021474s" totalImagesPullingTime="949.367025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:37.967470845 +0000 UTC m=+79.869927030" watchObservedRunningTime="2026-09-18 13:45:37.968021474 +0000 UTC m=+79.870477659"
Sep 18 13:45:38.959780 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:38.959743    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5mkl4" event={"ID":"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a","Type":"ContainerStarted","Data":"3f290cf8cf7f1187cc8fba1f9a462c05e89d36e41567e130edb02a5c6fc2288f"}
Sep 18 13:45:39.635310 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:39.635285    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:39.915105 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:39.915027    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8ksvp"
Sep 18 13:45:39.965798 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:39.965769    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5mkl4" event={"ID":"d4d7f5a1-3d5c-446d-b9df-8cf78eed185a","Type":"ContainerStarted","Data":"7239b4b938cdc7f478845f281e65b15ccc82917360e4cea80f4f00a0f7dfb1ff"}
Sep 18 13:45:39.985057 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:39.985005    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5mkl4" podStartSLOduration=33.242779078 podStartE2EDuration="34.984984277s" totalImagesPullingTime="1.742205199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:45:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:39.983651221 +0000 UTC m=+81.886107407" watchObservedRunningTime="2026-09-18 13:45:39.984984277 +0000 UTC m=+81.887440468"
Sep 18 13:45:46.093375 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.093342    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w6fd7"]
Sep 18 13:45:46.096465 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.096434    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.100351 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.100328    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 13:45:46.100623 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.100603    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 13:45:46.100835 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.100813    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 13:45:46.100936 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.100813    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 13:45:46.101145 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.101125    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 13:45:46.101911 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.101858    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 13:45:46.102024 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.102004    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vn8xp\""
Sep 18 13:45:46.206637 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.206615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.206773 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.206644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.206773 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.206664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-wtmp\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.206773 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.206691    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-metrics-client-ca\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.206773 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.206748    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-textfile\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.206979 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.206824    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-tls\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.206979 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.206877    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-sys\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.206979 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.206926    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-root\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.206979 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.206948    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw878\" (UniqueName: \"kubernetes.io/projected/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-kube-api-access-tw878\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.307909 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.307878    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-root\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.307909 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.307912    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw878\" (UniqueName: \"kubernetes.io/projected/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-kube-api-access-tw878\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308117 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.307938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308117 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.307951    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-root\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308117 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.307958    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308117 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.308009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-wtmp\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308117 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.308057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-metrics-client-ca\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308367 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.308241    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-textfile\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308367 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.308258    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-wtmp\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308367 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.308284    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-tls\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308367 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.308320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-sys\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308560 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.308400    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-sys\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.308560 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:46.308450    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 13:45:46.308560 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:46.308556    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-tls podName:dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6 nodeName:}" failed. No retries permitted until 2026-09-18 13:45:46.808536838 +0000 UTC m=+88.710993021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-tls") pod "node-exporter-w6fd7" (UID: "dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6") : secret "node-exporter-tls" not found
Sep 18 13:45:46.308996 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.308974    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.309137 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.309016    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-textfile\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.309137 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.309039    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-metrics-client-ca\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.311031 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.311013    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.318904 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.318881    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw878\" (UniqueName: \"kubernetes.io/projected/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-kube-api-access-tw878\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.812697 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.812667    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-tls\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:46.814803 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:46.814779    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6-node-exporter-tls\") pod \"node-exporter-w6fd7\" (UID: \"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6\") " pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:47.007252 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:47.007227    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w6fd7"
Sep 18 13:45:47.017562 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:45:47.017538    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbcf0f6_df17_4423_9018_d5c8dcd7e3e6.slice/crio-b306c8a561d92fceb2bd89d7284f71f45c215fbf14ccc3d2bba7f7c82957d3c2 WatchSource:0}: Error finding container b306c8a561d92fceb2bd89d7284f71f45c215fbf14ccc3d2bba7f7c82957d3c2: Status 404 returned error can't find the container with id b306c8a561d92fceb2bd89d7284f71f45c215fbf14ccc3d2bba7f7c82957d3c2
Sep 18 13:45:47.993521 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:47.993484    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6fd7" event={"ID":"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6","Type":"ContainerStarted","Data":"b16e7637ff6d10a4311641df498d41a8acefa72d7f559b2bd1bdf44d4aed70f0"}
Sep 18 13:45:47.993818 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:47.993596    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6fd7" event={"ID":"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6","Type":"ContainerStarted","Data":"b306c8a561d92fceb2bd89d7284f71f45c215fbf14ccc3d2bba7f7c82957d3c2"}
Sep 18 13:45:48.999656 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:48.999621    2569 generic.go:356] "Generic (PLEG): container finished" podID="dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6" containerID="b16e7637ff6d10a4311641df498d41a8acefa72d7f559b2bd1bdf44d4aed70f0" exitCode=0
Sep 18 13:45:49.000119 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:48.999701    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6fd7" event={"ID":"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6","Type":"ContainerDied","Data":"b16e7637ff6d10a4311641df498d41a8acefa72d7f559b2bd1bdf44d4aed70f0"}
Sep 18 13:45:50.005809 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:50.005778    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6fd7" event={"ID":"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6","Type":"ContainerStarted","Data":"e7f323355b2ceb7f8fc17e3a9a810cfad0180aaed61ad0f8d3869588ad370f19"}
Sep 18 13:45:50.006194 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:50.005828    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6fd7" event={"ID":"dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6","Type":"ContainerStarted","Data":"e6c5e6b735137cb2a77c185220882da3ec6b5d3fd33beafcff7e3dfb48ce44e0"}
Sep 18 13:45:50.038010 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:50.037968    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w6fd7" podStartSLOduration=3.133276051 podStartE2EDuration="4.03795475s" totalImagesPullingTime="837.572326ms" totalInitContainerRuntime="67.106373ms" isStatefulPod=true podCreationTimestamp="2026-09-18 13:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:50.035974588 +0000 UTC m=+91.938430798" watchObservedRunningTime="2026-09-18 13:45:50.03795475 +0000 UTC m=+91.940410962"
Sep 18 13:45:51.922285 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:51.922256    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7596bf68dd-lr7t4"
Sep 18 13:45:54.648310 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.648277    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4" podUID="a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" containerName="registry" containerID="cri-o://4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8" gracePeriod=30
Sep 18 13:45:54.888625 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.888606    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:54.967908 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.967886    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-image-registry-private-configuration\") pod \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") "
Sep 18 13:45:54.968055 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.967925    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls\") pod \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") "
Sep 18 13:45:54.968055 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.967969    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-certificates\") pod \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") "
Sep 18 13:45:54.968191 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.968103    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-ca-trust-extracted\") pod \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") "
Sep 18 13:45:54.968191 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.968154    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcwd2\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-kube-api-access-wcwd2\") pod \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") "
Sep 18 13:45:54.968288 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.968189    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-installation-pull-secrets\") pod \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") "
Sep 18 13:45:54.968288 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.968224    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-trusted-ca\") pod \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") "
Sep 18 13:45:54.968288 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.968253    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-bound-sa-token\") pod \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\" (UID: \"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54\") "
Sep 18 13:45:54.968438 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.968352    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:45:54.968488 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.968474    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-certificates\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:45:54.968787 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.968725    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:45:54.970601 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.970575    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:45:54.970601 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.970593    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:45:54.970895 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.970874    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-kube-api-access-wcwd2" (OuterVolumeSpecName: "kube-api-access-wcwd2") pod "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54"). InnerVolumeSpecName "kube-api-access-wcwd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:45:54.970940 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.970912    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:45:54.971166 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.971144    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:45:54.976624 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:54.976597    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" (UID: "a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:55.025411 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.025387    2569 generic.go:356] "Generic (PLEG): container finished" podID="a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" containerID="4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8" exitCode=0
Sep 18 13:45:55.025491 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.025422    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4" event={"ID":"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54","Type":"ContainerDied","Data":"4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8"}
Sep 18 13:45:55.025491 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.025437    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4"
Sep 18 13:45:55.025491 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.025480    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8d86cd8-7ctb4" event={"ID":"a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54","Type":"ContainerDied","Data":"1a40b797f9a2e11423ff0e4b6e629389db86786f59921b33a4b956da014efa22"}
Sep 18 13:45:55.025584 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.025508    2569 scope.go:117] "RemoveContainer" containerID="4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8"
Sep 18 13:45:55.033476 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.033459    2569 scope.go:117] "RemoveContainer" containerID="4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8"
Sep 18 13:45:55.033759 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:45:55.033737    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8\": container with ID starting with 4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8 not found: ID does not exist" containerID="4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8"
Sep 18 13:45:55.033824 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.033765    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8"} err="failed to get container status \"4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8\": rpc error: code = NotFound desc = could not find container \"4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8\": container with ID starting with 4141397eabc4f7f483764c888c31f0f42134c49a12436b875d206086fdc806c8 not found: ID does not exist"
Sep 18 13:45:55.047881 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.047863    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8d86cd8-7ctb4"]
Sep 18 13:45:55.052713 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.052693    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c8d86cd8-7ctb4"]
Sep 18 13:45:55.068752 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.068735    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-image-registry-private-configuration\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:45:55.068815 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.068754    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-registry-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:45:55.068815 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.068764    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-ca-trust-extracted\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:45:55.068815 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.068771    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcwd2\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-kube-api-access-wcwd2\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:45:55.068815 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.068780    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-installation-pull-secrets\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:45:55.068815 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.068787    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-trusted-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:45:55.068815 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:55.068795    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54-bound-sa-token\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:45:56.484323 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:56.484293    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" path="/var/lib/kubelet/pods/a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54/volumes"
Sep 18 13:45:59.903184 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:45:59.903161    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xsq2g"
Sep 18 13:46:30.143318 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:46:30.143291    2569 generic.go:356] "Generic (PLEG): container finished" podID="f031d87d-90bc-46f0-b1e1-9cc72ef38e3b" containerID="b24167b480b2399bb2cf800dd9930c9008b54f5a9c47897f5ce7ceb3fc928c99" exitCode=0
Sep 18 13:46:30.143702 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:46:30.143364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l" event={"ID":"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b","Type":"ContainerDied","Data":"b24167b480b2399bb2cf800dd9930c9008b54f5a9c47897f5ce7ceb3fc928c99"}
Sep 18 13:46:30.143702 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:46:30.143645    2569 scope.go:117] "RemoveContainer" containerID="b24167b480b2399bb2cf800dd9930c9008b54f5a9c47897f5ce7ceb3fc928c99"
Sep 18 13:46:31.150032 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:46:31.149996    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-9rn5l" event={"ID":"f031d87d-90bc-46f0-b1e1-9cc72ef38e3b","Type":"ContainerStarted","Data":"5b67598a73cc93d2613655080ec77bf0cefe536fba67e2c348ae8e8e613ff342"}
Sep 18 13:46:40.182505 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:46:40.182478    2569 generic.go:356] "Generic (PLEG): container finished" podID="cc6c4e02-80c7-463b-8cca-be8f1588caee" containerID="2e1ba94b7f300479dfa6e68ef7cb6af0442b8d516f8f7dab1c8187293bb81f34" exitCode=0
Sep 18 13:46:40.182934 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:46:40.182526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp" event={"ID":"cc6c4e02-80c7-463b-8cca-be8f1588caee","Type":"ContainerDied","Data":"2e1ba94b7f300479dfa6e68ef7cb6af0442b8d516f8f7dab1c8187293bb81f34"}
Sep 18 13:46:40.182934 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:46:40.182877    2569 scope.go:117] "RemoveContainer" containerID="2e1ba94b7f300479dfa6e68ef7cb6af0442b8d516f8f7dab1c8187293bb81f34"
Sep 18 13:46:41.188108 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:46:41.188059    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-sfzlp" event={"ID":"cc6c4e02-80c7-463b-8cca-be8f1588caee","Type":"ContainerStarted","Data":"40aad484d55a6ba96de0f11ad1c43cefcd09184755a9b69bb767b0c1789846c7"}
Sep 18 13:49:18.423214 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:18.423175    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:49:18.428385 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:18.428368    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:49:18.428982 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:18.428966    2569 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 13:49:24.002179 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.002149    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"]
Sep 18 13:49:24.004339 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.002399    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" containerName="registry"
Sep 18 13:49:24.004339 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.002409    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" containerName="registry"
Sep 18 13:49:24.004339 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.002460    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ac64cd-1f2f-4cd2-b651-4ef63ef14c54" containerName="registry"
Sep 18 13:49:24.005184 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.005170    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.010719 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.010699    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 13:49:24.010833 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.010738    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dlrjx\""
Sep 18 13:49:24.010913 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.010902    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 13:49:24.011289 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.011277    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 13:49:24.011364 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.011346    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 13:49:24.032784 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.032765    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"]
Sep 18 13:49:24.044987 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.044958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d848ed91-7685-4d82-a7cc-2717d05a00d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.045120 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.045013    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d848ed91-7685-4d82-a7cc-2717d05a00d1-manifests\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.045165 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.045127    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmphq\" (UniqueName: \"kubernetes.io/projected/d848ed91-7685-4d82-a7cc-2717d05a00d1-kube-api-access-jmphq\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.045165 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.045154    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d848ed91-7685-4d82-a7cc-2717d05a00d1-tmp\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.045230 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.045185    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d848ed91-7685-4d82-a7cc-2717d05a00d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.145721 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.145683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmphq\" (UniqueName: \"kubernetes.io/projected/d848ed91-7685-4d82-a7cc-2717d05a00d1-kube-api-access-jmphq\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.145721 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.145724    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d848ed91-7685-4d82-a7cc-2717d05a00d1-tmp\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.145904 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.145744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d848ed91-7685-4d82-a7cc-2717d05a00d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.145904 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.145772    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d848ed91-7685-4d82-a7cc-2717d05a00d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.145904 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.145792    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d848ed91-7685-4d82-a7cc-2717d05a00d1-manifests\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.146151 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.146137    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d848ed91-7685-4d82-a7cc-2717d05a00d1-manifests\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.147961 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.147940    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d848ed91-7685-4d82-a7cc-2717d05a00d1-tmp\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.148248 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.148230    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d848ed91-7685-4d82-a7cc-2717d05a00d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.148364 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.148344    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d848ed91-7685-4d82-a7cc-2717d05a00d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.155028 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.155006    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmphq\" (UniqueName: \"kubernetes.io/projected/d848ed91-7685-4d82-a7cc-2717d05a00d1-kube-api-access-jmphq\") pod \"opendatahub-operator-controller-manager-978ccd68d-j6jwg\" (UID: \"d848ed91-7685-4d82-a7cc-2717d05a00d1\") " pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.314880 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.314805    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:24.591447 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.591422    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"]
Sep 18 13:49:24.593248 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:49:24.593214    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd848ed91_7685_4d82_a7cc_2717d05a00d1.slice/crio-06ec6b0afc8aad07da08a5dfea83f1a5549af33a6bd713414f565a4d896fceb2 WatchSource:0}: Error finding container 06ec6b0afc8aad07da08a5dfea83f1a5549af33a6bd713414f565a4d896fceb2: Status 404 returned error can't find the container with id 06ec6b0afc8aad07da08a5dfea83f1a5549af33a6bd713414f565a4d896fceb2
Sep 18 13:49:24.594787 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.594774    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:49:24.734980 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:24.734951    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg" event={"ID":"d848ed91-7685-4d82-a7cc-2717d05a00d1","Type":"ContainerStarted","Data":"06ec6b0afc8aad07da08a5dfea83f1a5549af33a6bd713414f565a4d896fceb2"}
Sep 18 13:49:27.750280 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:27.750219    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg" event={"ID":"d848ed91-7685-4d82-a7cc-2717d05a00d1","Type":"ContainerStarted","Data":"09cf5bc6d870216b0adf494a129919066dfdf4b61eddbf1796daa74ee0e8bd3d"}
Sep 18 13:49:28.754658 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:28.754634    2569 generic.go:356] "Generic (PLEG): container finished" podID="d848ed91-7685-4d82-a7cc-2717d05a00d1" containerID="09cf5bc6d870216b0adf494a129919066dfdf4b61eddbf1796daa74ee0e8bd3d" exitCode=0
Sep 18 13:49:28.754972 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:28.754665    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg" event={"ID":"d848ed91-7685-4d82-a7cc-2717d05a00d1","Type":"ContainerDied","Data":"09cf5bc6d870216b0adf494a129919066dfdf4b61eddbf1796daa74ee0e8bd3d"}
Sep 18 13:49:29.760789 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:29.760753    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg" event={"ID":"d848ed91-7685-4d82-a7cc-2717d05a00d1","Type":"ContainerStarted","Data":"b1a670a15d1fae672d170f1a4acd8f58c76fd36a8b0bbed34b9ae75417cbf5bc"}
Sep 18 13:49:29.761188 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:29.760882    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:29.812165 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:29.812109    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg" podStartSLOduration=2.615344703 podStartE2EDuration="6.812089436s" totalImagesPullingTime="3.202321532s" totalInitContainerRuntime="994.423201ms" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:29.809981051 +0000 UTC m=+311.712437227" watchObservedRunningTime="2026-09-18 13:49:29.812089436 +0000 UTC m=+311.714545614"
Sep 18 13:49:40.765921 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:40.765895    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-978ccd68d-j6jwg"
Sep 18 13:49:43.460234 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.460208    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"]
Sep 18 13:49:43.465849 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.465826    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:43.469642 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.469620    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 13:49:43.469737 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.469698    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 13:49:43.469737 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.469698    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-t2mk4\""
Sep 18 13:49:43.480683 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.480661    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"]
Sep 18 13:49:43.581512 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.581489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gps5k\" (UniqueName: \"kubernetes.io/projected/4e71f7b4-32a8-4775-894c-a9853de09d38-kube-api-access-gps5k\") pod \"kube-auth-proxy-d5c57f774-9wsc8\" (UID: \"4e71f7b4-32a8-4775-894c-a9853de09d38\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:43.581626 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.581535    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e71f7b4-32a8-4775-894c-a9853de09d38-tls-certs\") pod \"kube-auth-proxy-d5c57f774-9wsc8\" (UID: \"4e71f7b4-32a8-4775-894c-a9853de09d38\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:43.581626 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.581574    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e71f7b4-32a8-4775-894c-a9853de09d38-tmp\") pod \"kube-auth-proxy-d5c57f774-9wsc8\" (UID: \"4e71f7b4-32a8-4775-894c-a9853de09d38\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:43.682740 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.682712    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gps5k\" (UniqueName: \"kubernetes.io/projected/4e71f7b4-32a8-4775-894c-a9853de09d38-kube-api-access-gps5k\") pod \"kube-auth-proxy-d5c57f774-9wsc8\" (UID: \"4e71f7b4-32a8-4775-894c-a9853de09d38\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:43.682877 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.682763    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e71f7b4-32a8-4775-894c-a9853de09d38-tls-certs\") pod \"kube-auth-proxy-d5c57f774-9wsc8\" (UID: \"4e71f7b4-32a8-4775-894c-a9853de09d38\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:43.682877 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.682788    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e71f7b4-32a8-4775-894c-a9853de09d38-tmp\") pod \"kube-auth-proxy-d5c57f774-9wsc8\" (UID: \"4e71f7b4-32a8-4775-894c-a9853de09d38\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:43.685226 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.685199    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e71f7b4-32a8-4775-894c-a9853de09d38-tmp\") pod \"kube-auth-proxy-d5c57f774-9wsc8\" (UID: \"4e71f7b4-32a8-4775-894c-a9853de09d38\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:43.685449 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.685431    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e71f7b4-32a8-4775-894c-a9853de09d38-tls-certs\") pod \"kube-auth-proxy-d5c57f774-9wsc8\" (UID: \"4e71f7b4-32a8-4775-894c-a9853de09d38\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:43.693690 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.693669    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gps5k\" (UniqueName: \"kubernetes.io/projected/4e71f7b4-32a8-4775-894c-a9853de09d38-kube-api-access-gps5k\") pod \"kube-auth-proxy-d5c57f774-9wsc8\" (UID: \"4e71f7b4-32a8-4775-894c-a9853de09d38\") " pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:43.775873 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:43.775823    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"
Sep 18 13:49:44.051503 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:44.051479    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8"]
Sep 18 13:49:44.053737 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:49:44.053705    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e71f7b4_32a8_4775_894c_a9853de09d38.slice/crio-922361f8f2164d40b0a848cb6a5ad50142958e4d76fcaa25ed9d58e0116f9809 WatchSource:0}: Error finding container 922361f8f2164d40b0a848cb6a5ad50142958e4d76fcaa25ed9d58e0116f9809: Status 404 returned error can't find the container with id 922361f8f2164d40b0a848cb6a5ad50142958e4d76fcaa25ed9d58e0116f9809
Sep 18 13:49:44.812138 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:44.812107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8" event={"ID":"4e71f7b4-32a8-4775-894c-a9853de09d38","Type":"ContainerStarted","Data":"922361f8f2164d40b0a848cb6a5ad50142958e4d76fcaa25ed9d58e0116f9809"}
Sep 18 13:49:47.827091 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:47.827031    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8" event={"ID":"4e71f7b4-32a8-4775-894c-a9853de09d38","Type":"ContainerStarted","Data":"8842f73c4c9b7df681d15477cd0e9069b03780b1b4b82dbe574d2ab6719c95ce"}
Sep 18 13:49:47.845622 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:47.845559    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d5c57f774-9wsc8" podStartSLOduration=2.043336873 podStartE2EDuration="4.845544587s" totalImagesPullingTime="2.802207714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:47.843979863 +0000 UTC m=+329.746436047" watchObservedRunningTime="2026-09-18 13:49:47.845544587 +0000 UTC m=+329.748000810"
Sep 18 13:49:54.993869 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:54.993840    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xttwp"]
Sep 18 13:49:54.996583 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:54.996567    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:49:54.999611 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:54.999589    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 18 13:49:55.010253 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:55.010233    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xttwp"]
Sep 18 13:49:55.072854 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:55.072822    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvbf\" (UniqueName: \"kubernetes.io/projected/eb9f7b40-29d0-4d27-8b52-b1e14cd41efd-kube-api-access-vtvbf\") pod \"model-serving-api-54d89bd5fc-xttwp\" (UID: \"eb9f7b40-29d0-4d27-8b52-b1e14cd41efd\") " pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:49:55.073002 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:55.072914    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9f7b40-29d0-4d27-8b52-b1e14cd41efd-tls-certs\") pod \"model-serving-api-54d89bd5fc-xttwp\" (UID: \"eb9f7b40-29d0-4d27-8b52-b1e14cd41efd\") " pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:49:55.173973 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:55.173937    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9f7b40-29d0-4d27-8b52-b1e14cd41efd-tls-certs\") pod \"model-serving-api-54d89bd5fc-xttwp\" (UID: \"eb9f7b40-29d0-4d27-8b52-b1e14cd41efd\") " pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:49:55.174169 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:55.173992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvbf\" (UniqueName: \"kubernetes.io/projected/eb9f7b40-29d0-4d27-8b52-b1e14cd41efd-kube-api-access-vtvbf\") pod \"model-serving-api-54d89bd5fc-xttwp\" (UID: \"eb9f7b40-29d0-4d27-8b52-b1e14cd41efd\") " pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:49:55.174169 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:49:55.174135    2569 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 18 13:49:55.174254 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:49:55.174223    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9f7b40-29d0-4d27-8b52-b1e14cd41efd-tls-certs podName:eb9f7b40-29d0-4d27-8b52-b1e14cd41efd nodeName:}" failed. No retries permitted until 2026-09-18 13:49:55.674201317 +0000 UTC m=+337.576657486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/eb9f7b40-29d0-4d27-8b52-b1e14cd41efd-tls-certs") pod "model-serving-api-54d89bd5fc-xttwp" (UID: "eb9f7b40-29d0-4d27-8b52-b1e14cd41efd") : secret "model-serving-api-tls" not found
Sep 18 13:49:55.188405 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:55.188373    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvbf\" (UniqueName: \"kubernetes.io/projected/eb9f7b40-29d0-4d27-8b52-b1e14cd41efd-kube-api-access-vtvbf\") pod \"model-serving-api-54d89bd5fc-xttwp\" (UID: \"eb9f7b40-29d0-4d27-8b52-b1e14cd41efd\") " pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:49:55.677701 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:55.677663    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9f7b40-29d0-4d27-8b52-b1e14cd41efd-tls-certs\") pod \"model-serving-api-54d89bd5fc-xttwp\" (UID: \"eb9f7b40-29d0-4d27-8b52-b1e14cd41efd\") " pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:49:55.680442 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:55.680413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9f7b40-29d0-4d27-8b52-b1e14cd41efd-tls-certs\") pod \"model-serving-api-54d89bd5fc-xttwp\" (UID: \"eb9f7b40-29d0-4d27-8b52-b1e14cd41efd\") " pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:49:55.905833 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:55.905797    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:49:56.179734 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:56.179707    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xttwp"]
Sep 18 13:49:56.182297 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:49:56.182255    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9f7b40_29d0_4d27_8b52_b1e14cd41efd.slice/crio-b4361a565f19a5f3355b0028fe5df5361099f9876055c00841a7cbc31177ef2a WatchSource:0}: Error finding container b4361a565f19a5f3355b0028fe5df5361099f9876055c00841a7cbc31177ef2a: Status 404 returned error can't find the container with id b4361a565f19a5f3355b0028fe5df5361099f9876055c00841a7cbc31177ef2a
Sep 18 13:49:56.860798 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:56.860754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xttwp" event={"ID":"eb9f7b40-29d0-4d27-8b52-b1e14cd41efd","Type":"ContainerStarted","Data":"b4361a565f19a5f3355b0028fe5df5361099f9876055c00841a7cbc31177ef2a"}
Sep 18 13:49:57.866991 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:57.866949    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xttwp" event={"ID":"eb9f7b40-29d0-4d27-8b52-b1e14cd41efd","Type":"ContainerStarted","Data":"89204f8fed2f0b05c5bb5546f62cfd911bdcca65997a6bd5e4cdbc0672fd40e9"}
Sep 18 13:49:57.867483 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:57.867089    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:49:57.885875 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:49:57.885805    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-xttwp" podStartSLOduration=2.388177837 podStartE2EDuration="3.885786857s" totalImagesPullingTime="1.49760902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:57.884191239 +0000 UTC m=+339.786647460" watchObservedRunningTime="2026-09-18 13:49:57.885786857 +0000 UTC m=+339.788243042"
Sep 18 13:50:01.438102 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.438061    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"]
Sep 18 13:50:01.440268 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.440253    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.443717 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.443692    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tls6z\""
Sep 18 13:50:01.443848 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.443744    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 13:50:01.443848 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.443756    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 13:50:01.443939 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.443871    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 13:50:01.444291 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.444277    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 13:50:01.459824 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.459805    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"]
Sep 18 13:50:01.526555 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.526530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.526673 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.526559    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.526673 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.526584    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.526673 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.526625    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.526673 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.526666    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjff9\" (UniqueName: \"kubernetes.io/projected/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-kube-api-access-sjff9\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.526835 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.526710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.526835 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.526738    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.627908 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.627877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.628055 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.627914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.628055 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.627972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.628470 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.628196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.628539 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.628519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjff9\" (UniqueName: \"kubernetes.io/projected/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-kube-api-access-sjff9\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.628585 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.628560    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.628637 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.628595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.628833 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.628814    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.630525 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.630501    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.630636 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.630615    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.630912 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.630893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.630986 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.630951    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.646064 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.646042    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjff9\" (UniqueName: \"kubernetes.io/projected/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-kube-api-access-sjff9\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.652811 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.652789    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9lvdf\" (UID: \"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:01.749287 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:01.749259    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:02.030369 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:02.030274    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"]
Sep 18 13:50:02.033267 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:50:02.033235    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6a0d3c_03cc_4ee1_893e_4d2ce90d5ab7.slice/crio-5e8a957899103b2e03737c83e0e44cf30755048da68087f10de79dcfdc658307 WatchSource:0}: Error finding container 5e8a957899103b2e03737c83e0e44cf30755048da68087f10de79dcfdc658307: Status 404 returned error can't find the container with id 5e8a957899103b2e03737c83e0e44cf30755048da68087f10de79dcfdc658307
Sep 18 13:50:02.886265 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:02.886234    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf" event={"ID":"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7","Type":"ContainerStarted","Data":"5e8a957899103b2e03737c83e0e44cf30755048da68087f10de79dcfdc658307"}
Sep 18 13:50:05.030661 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:05.030617    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:50:05.031016 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:05.030711    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:50:05.902801 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:05.902760    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf" event={"ID":"3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7","Type":"ContainerStarted","Data":"522f9b665dca0298b8d505b31d192c7ed067bd40bc94874f0c4c483452fbba4f"}
Sep 18 13:50:05.903043 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:05.902985    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:05.928350 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:05.928289    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf" podStartSLOduration=1.933849512 podStartE2EDuration="4.928272453s" totalImagesPullingTime="2.994422941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:50:05.926605393 +0000 UTC m=+347.829061592" watchObservedRunningTime="2026-09-18 13:50:05.928272453 +0000 UTC m=+347.830728637"
Sep 18 13:50:06.908665 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:06.908640    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9lvdf"
Sep 18 13:50:08.875091 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:50:08.875047    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-xttwp"
Sep 18 13:51:34.267293 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.267265    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"]
Sep 18 13:51:34.270379 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.270365    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:34.273562 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.273547    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 13:51:34.274680 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.274667    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 13:51:34.275856 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.275838    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bcr8n\""
Sep 18 13:51:34.290380 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.290353    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"]
Sep 18 13:51:34.400108 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.400065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4dm\" (UniqueName: \"kubernetes.io/projected/ecfe4068-3411-4805-8847-cbb1e7ad1afc-kube-api-access-2z4dm\") pod \"limitador-operator-controller-manager-74495c69f8-j5mgh\" (UID: \"ecfe4068-3411-4805-8847-cbb1e7ad1afc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:34.501263 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.501236    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4dm\" (UniqueName: \"kubernetes.io/projected/ecfe4068-3411-4805-8847-cbb1e7ad1afc-kube-api-access-2z4dm\") pod \"limitador-operator-controller-manager-74495c69f8-j5mgh\" (UID: \"ecfe4068-3411-4805-8847-cbb1e7ad1afc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:34.520782 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.520725    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4dm\" (UniqueName: \"kubernetes.io/projected/ecfe4068-3411-4805-8847-cbb1e7ad1afc-kube-api-access-2z4dm\") pod \"limitador-operator-controller-manager-74495c69f8-j5mgh\" (UID: \"ecfe4068-3411-4805-8847-cbb1e7ad1afc\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:34.579668 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.579641    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:34.859597 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:34.859544    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"]
Sep 18 13:51:34.862337 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:51:34.862307    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfe4068_3411_4805_8847_cbb1e7ad1afc.slice/crio-62cc2bc4c239862b5e874c3713334fbe90c4f36cbaae7207f788f7a24168ff33 WatchSource:0}: Error finding container 62cc2bc4c239862b5e874c3713334fbe90c4f36cbaae7207f788f7a24168ff33: Status 404 returned error can't find the container with id 62cc2bc4c239862b5e874c3713334fbe90c4f36cbaae7207f788f7a24168ff33
Sep 18 13:51:35.257963 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:35.257927    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh" event={"ID":"ecfe4068-3411-4805-8847-cbb1e7ad1afc","Type":"ContainerStarted","Data":"62cc2bc4c239862b5e874c3713334fbe90c4f36cbaae7207f788f7a24168ff33"}
Sep 18 13:51:36.365218 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:36.365190    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv"]
Sep 18 13:51:36.371995 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:36.371975    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv"
Sep 18 13:51:36.374757 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:36.374736    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 13:51:36.374870 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:36.374736    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jvmg5\""
Sep 18 13:51:36.380458 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:36.380382    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv"]
Sep 18 13:51:36.516683 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:36.516652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpjqx\" (UniqueName: \"kubernetes.io/projected/db99daa1-02fc-4b79-be6a-132f09da2ed1-kube-api-access-fpjqx\") pod \"dns-operator-controller-manager-7b84cd7c9c-dhgtv\" (UID: \"db99daa1-02fc-4b79-be6a-132f09da2ed1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv"
Sep 18 13:51:36.618035 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:36.617946    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjqx\" (UniqueName: \"kubernetes.io/projected/db99daa1-02fc-4b79-be6a-132f09da2ed1-kube-api-access-fpjqx\") pod \"dns-operator-controller-manager-7b84cd7c9c-dhgtv\" (UID: \"db99daa1-02fc-4b79-be6a-132f09da2ed1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv"
Sep 18 13:51:36.633658 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:36.633630    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpjqx\" (UniqueName: \"kubernetes.io/projected/db99daa1-02fc-4b79-be6a-132f09da2ed1-kube-api-access-fpjqx\") pod \"dns-operator-controller-manager-7b84cd7c9c-dhgtv\" (UID: \"db99daa1-02fc-4b79-be6a-132f09da2ed1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv"
Sep 18 13:51:36.685456 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:36.685426    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv"
Sep 18 13:51:37.495800 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:37.495720    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv"]
Sep 18 13:51:37.499717 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:51:37.499691    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb99daa1_02fc_4b79_be6a_132f09da2ed1.slice/crio-20e4145011e5f1ecef509dac4e3a66f9bc415d5e29fe916a4d4f06fcff81f5c0 WatchSource:0}: Error finding container 20e4145011e5f1ecef509dac4e3a66f9bc415d5e29fe916a4d4f06fcff81f5c0: Status 404 returned error can't find the container with id 20e4145011e5f1ecef509dac4e3a66f9bc415d5e29fe916a4d4f06fcff81f5c0
Sep 18 13:51:38.272865 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:38.272828    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv" event={"ID":"db99daa1-02fc-4b79-be6a-132f09da2ed1","Type":"ContainerStarted","Data":"20e4145011e5f1ecef509dac4e3a66f9bc415d5e29fe916a4d4f06fcff81f5c0"}
Sep 18 13:51:38.275053 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:38.275018    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh" event={"ID":"ecfe4068-3411-4805-8847-cbb1e7ad1afc","Type":"ContainerStarted","Data":"fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d"}
Sep 18 13:51:38.275286 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:38.275262    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:38.305432 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:38.305379    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh" podStartSLOduration=1.8989348700000002 podStartE2EDuration="4.305193209s" totalImagesPullingTime="2.406258339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:51:38.303894782 +0000 UTC m=+440.206350968" watchObservedRunningTime="2026-09-18 13:51:38.305193209 +0000 UTC m=+440.207649395"
Sep 18 13:51:39.243708 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.243675    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-rtn8p"]
Sep 18 13:51:39.246992 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.246976    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-rtn8p"
Sep 18 13:51:39.250489 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.250467    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-v84c4\""
Sep 18 13:51:39.272289 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.272267    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-rtn8p"]
Sep 18 13:51:39.281042 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.281020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv" event={"ID":"db99daa1-02fc-4b79-be6a-132f09da2ed1","Type":"ContainerStarted","Data":"5b0529d954d3a531e6a0364db2715bb45dc5588bfae33d0fe7a4135330b45b8b"}
Sep 18 13:51:39.281212 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.281195    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv"
Sep 18 13:51:39.340680 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.340619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5wqw\" (UniqueName: \"kubernetes.io/projected/0390bcd3-c90d-40c7-951f-f7fd43e2dda4-kube-api-access-n5wqw\") pod \"authorino-operator-76b478bd78-rtn8p\" (UID: \"0390bcd3-c90d-40c7-951f-f7fd43e2dda4\") " pod="kuadrant-system/authorino-operator-76b478bd78-rtn8p"
Sep 18 13:51:39.392494 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.392450    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv" podStartSLOduration=1.816409242 podStartE2EDuration="3.39243592s" totalImagesPullingTime="1.576026678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:51:39.390418904 +0000 UTC m=+441.292875089" watchObservedRunningTime="2026-09-18 13:51:39.39243592 +0000 UTC m=+441.294892105"
Sep 18 13:51:39.441236 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.441209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5wqw\" (UniqueName: \"kubernetes.io/projected/0390bcd3-c90d-40c7-951f-f7fd43e2dda4-kube-api-access-n5wqw\") pod \"authorino-operator-76b478bd78-rtn8p\" (UID: \"0390bcd3-c90d-40c7-951f-f7fd43e2dda4\") " pod="kuadrant-system/authorino-operator-76b478bd78-rtn8p"
Sep 18 13:51:39.453422 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.453402    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5wqw\" (UniqueName: \"kubernetes.io/projected/0390bcd3-c90d-40c7-951f-f7fd43e2dda4-kube-api-access-n5wqw\") pod \"authorino-operator-76b478bd78-rtn8p\" (UID: \"0390bcd3-c90d-40c7-951f-f7fd43e2dda4\") " pod="kuadrant-system/authorino-operator-76b478bd78-rtn8p"
Sep 18 13:51:39.557100 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.557061    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-rtn8p"
Sep 18 13:51:39.834242 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:39.834155    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-rtn8p"]
Sep 18 13:51:39.836882 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:51:39.836844    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0390bcd3_c90d_40c7_951f_f7fd43e2dda4.slice/crio-fc1c2aa05ab0ae0e323e9ff198ad86785b09bc054b3f7bea5b044e38229f7611 WatchSource:0}: Error finding container fc1c2aa05ab0ae0e323e9ff198ad86785b09bc054b3f7bea5b044e38229f7611: Status 404 returned error can't find the container with id fc1c2aa05ab0ae0e323e9ff198ad86785b09bc054b3f7bea5b044e38229f7611
Sep 18 13:51:40.285857 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:40.285784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-rtn8p" event={"ID":"0390bcd3-c90d-40c7-951f-f7fd43e2dda4","Type":"ContainerStarted","Data":"fc1c2aa05ab0ae0e323e9ff198ad86785b09bc054b3f7bea5b044e38229f7611"}
Sep 18 13:51:42.295893 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:42.295862    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-rtn8p" event={"ID":"0390bcd3-c90d-40c7-951f-f7fd43e2dda4","Type":"ContainerStarted","Data":"85fec331158d97a1e273fad2723dcc687487c435efa6d82415e1e32c9554f597"}
Sep 18 13:51:42.296299 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:42.295974    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-rtn8p"
Sep 18 13:51:42.314569 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:42.314527    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-rtn8p" podStartSLOduration=1.7686487770000001 podStartE2EDuration="3.314513329s" totalImagesPullingTime="1.545864552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:51:42.312960171 +0000 UTC m=+444.215416357" watchObservedRunningTime="2026-09-18 13:51:42.314513329 +0000 UTC m=+444.216969513"
Sep 18 13:51:49.284052 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:49.284020    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:50.288416 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.288388    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-dhgtv"
Sep 18 13:51:50.705983 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.705951    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"]
Sep 18 13:51:50.706310 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.706257    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh" podUID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" containerName="manager" containerID="cri-o://fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d" gracePeriod=2
Sep 18 13:51:50.721753 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.721729    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"]
Sep 18 13:51:50.725619 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:51:50.725585    2569 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ecfe4068-3411-4805-8847-cbb1e7ad1afc\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-j5mgh\": pods \"limitador-operator-controller-manager-74495c69f8-j5mgh\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:50.734374 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.734352    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk"]
Sep 18 13:51:50.734637 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.734626    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" containerName="manager"
Sep 18 13:51:50.734677 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.734639    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" containerName="manager"
Sep 18 13:51:50.734709 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.734693    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" containerName="manager"
Sep 18 13:51:50.737567 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.737550    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk"
Sep 18 13:51:50.758889 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.758858    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk"]
Sep 18 13:51:50.789140 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:51:50.789111    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-j5mgh\" is forbidden: User \"system:node:ip-10-0-143-135.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-135.ec2.internal' and this object" podUID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:50.926222 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.926195    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42whm\" (UniqueName: \"kubernetes.io/projected/24846b11-1e3d-44c2-aac6-81279410b544-kube-api-access-42whm\") pod \"limitador-operator-controller-manager-74495c69f8-6smsk\" (UID: \"24846b11-1e3d-44c2-aac6-81279410b544\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk"
Sep 18 13:51:50.938891 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:50.938872    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:50.941639 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:51:50.941615    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-j5mgh\" is forbidden: User \"system:node:ip-10-0-143-135.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-135.ec2.internal' and this object" podUID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:51.026813 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.026756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42whm\" (UniqueName: \"kubernetes.io/projected/24846b11-1e3d-44c2-aac6-81279410b544-kube-api-access-42whm\") pod \"limitador-operator-controller-manager-74495c69f8-6smsk\" (UID: \"24846b11-1e3d-44c2-aac6-81279410b544\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk"
Sep 18 13:51:51.036966 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.036944    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42whm\" (UniqueName: \"kubernetes.io/projected/24846b11-1e3d-44c2-aac6-81279410b544-kube-api-access-42whm\") pod \"limitador-operator-controller-manager-74495c69f8-6smsk\" (UID: \"24846b11-1e3d-44c2-aac6-81279410b544\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk"
Sep 18 13:51:51.088285 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.088264    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk"
Sep 18 13:51:51.127082 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.127044    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z4dm\" (UniqueName: \"kubernetes.io/projected/ecfe4068-3411-4805-8847-cbb1e7ad1afc-kube-api-access-2z4dm\") pod \"ecfe4068-3411-4805-8847-cbb1e7ad1afc\" (UID: \"ecfe4068-3411-4805-8847-cbb1e7ad1afc\") "
Sep 18 13:51:51.129313 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.129285    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfe4068-3411-4805-8847-cbb1e7ad1afc-kube-api-access-2z4dm" (OuterVolumeSpecName: "kube-api-access-2z4dm") pod "ecfe4068-3411-4805-8847-cbb1e7ad1afc" (UID: "ecfe4068-3411-4805-8847-cbb1e7ad1afc"). InnerVolumeSpecName "kube-api-access-2z4dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:51.228586 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.228557    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z4dm\" (UniqueName: \"kubernetes.io/projected/ecfe4068-3411-4805-8847-cbb1e7ad1afc-kube-api-access-2z4dm\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:51:51.333695 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.333606    2569 generic.go:356] "Generic (PLEG): container finished" podID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" containerID="fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d" exitCode=0
Sep 18 13:51:51.333695 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.333654    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:51.334166 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.333712    2569 scope.go:117] "RemoveContainer" containerID="fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d"
Sep 18 13:51:51.337020 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:51:51.336989    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-j5mgh\" is forbidden: User \"system:node:ip-10-0-143-135.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-135.ec2.internal' and this object" podUID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:51.342776 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.342760    2569 scope.go:117] "RemoveContainer" containerID="fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d"
Sep 18 13:51:51.343031 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:51:51.343013    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d\": container with ID starting with fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d not found: ID does not exist" containerID="fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d"
Sep 18 13:51:51.343088 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.343042    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d"} err="failed to get container status \"fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d\": rpc error: code = NotFound desc = could not find container \"fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d\": container with ID starting with fd2f04bc6ae74f4f0d77b98f9dae79b2da676ba16b5e70b535647032bb592d7d not found: ID does not exist"
Sep 18 13:51:51.345820 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:51:51.345800    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-j5mgh\" is forbidden: User \"system:node:ip-10-0-143-135.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-135.ec2.internal' and this object" podUID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:51.376759 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:51.376738    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk"]
Sep 18 13:51:51.377633 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:51:51.377598    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24846b11_1e3d_44c2_aac6_81279410b544.slice/crio-69564499ec923d33810cbbdbd3e0e4d663d9ec041d250a1319413fae2230edf2 WatchSource:0}: Error finding container 69564499ec923d33810cbbdbd3e0e4d663d9ec041d250a1319413fae2230edf2: Status 404 returned error can't find the container with id 69564499ec923d33810cbbdbd3e0e4d663d9ec041d250a1319413fae2230edf2
Sep 18 13:51:52.339502 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:52.339467    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk" event={"ID":"24846b11-1e3d-44c2-aac6-81279410b544","Type":"ContainerStarted","Data":"1527894263f94a1d05b5e937fde3ce11b62bb1d28c8be164789f51a1b93010e5"}
Sep 18 13:51:52.339968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:52.339523    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk" event={"ID":"24846b11-1e3d-44c2-aac6-81279410b544","Type":"ContainerStarted","Data":"69564499ec923d33810cbbdbd3e0e4d663d9ec041d250a1319413fae2230edf2"}
Sep 18 13:51:52.339968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:52.339571    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk"
Sep 18 13:51:52.361341 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:52.361291    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk" podStartSLOduration=2.361277341 podStartE2EDuration="2.361277341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:51:52.359493004 +0000 UTC m=+454.261949189" watchObservedRunningTime="2026-09-18 13:51:52.361277341 +0000 UTC m=+454.263733525"
Sep 18 13:51:52.362276 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:51:52.362249    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-j5mgh\" is forbidden: User \"system:node:ip-10-0-143-135.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-135.ec2.internal' and this object" podUID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j5mgh"
Sep 18 13:51:52.485487 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:52.485455    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecfe4068-3411-4805-8847-cbb1e7ad1afc" path="/var/lib/kubelet/pods/ecfe4068-3411-4805-8847-cbb1e7ad1afc/volumes"
Sep 18 13:51:53.302107 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:51:53.302061    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-rtn8p"
Sep 18 13:52:03.346414 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:52:03.346382    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6smsk"
Sep 18 13:53:06.025126 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.025058    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57ddb66d95-skpmb"]
Sep 18 13:53:06.028778 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.028760    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.031874 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.031702    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9kgv2\""
Sep 18 13:53:06.031874 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.031714    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 13:53:06.031874 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.031704    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 13:53:06.031874 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.031751    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 13:53:06.040649 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.040629    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57ddb66d95-skpmb"]
Sep 18 13:53:06.154352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.154323    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2wxk\" (UniqueName: \"kubernetes.io/projected/eff86707-d908-4581-b818-604719733eb3-kube-api-access-m2wxk\") pod \"maas-controller-57ddb66d95-skpmb\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") " pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.154352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.154356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-webhook-cert\") pod \"maas-controller-57ddb66d95-skpmb\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") " pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.154553 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.154404    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-metrics-certs\") pod \"maas-controller-57ddb66d95-skpmb\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") " pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.238974 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.238947    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bd575b874-dd4ht"]
Sep 18 13:53:06.241998 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.241984    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:06.253405 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.253383    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bd575b874-dd4ht"]
Sep 18 13:53:06.255248 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.255227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-webhook-cert\") pod \"maas-controller-57ddb66d95-skpmb\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") " pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.255344 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.255315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-metrics-certs\") pod \"maas-controller-57ddb66d95-skpmb\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") " pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.255403 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.255362    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2wxk\" (UniqueName: \"kubernetes.io/projected/eff86707-d908-4581-b818-604719733eb3-kube-api-access-m2wxk\") pod \"maas-controller-57ddb66d95-skpmb\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") " pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.255403 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:53:06.255394    2569 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 18 13:53:06.255506 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:53:06.255465    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-webhook-cert podName:eff86707-d908-4581-b818-604719733eb3 nodeName:}" failed. No retries permitted until 2026-09-18 13:53:06.755445799 +0000 UTC m=+528.657901976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-webhook-cert") pod "maas-controller-57ddb66d95-skpmb" (UID: "eff86707-d908-4581-b818-604719733eb3") : secret "maas-controller-webhook-cert" not found
Sep 18 13:53:06.257890 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.257871    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-metrics-certs\") pod \"maas-controller-57ddb66d95-skpmb\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") " pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.269118 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.269092    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2wxk\" (UniqueName: \"kubernetes.io/projected/eff86707-d908-4581-b818-604719733eb3-kube-api-access-m2wxk\") pod \"maas-controller-57ddb66d95-skpmb\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") " pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.355959 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.355902    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-webhook-cert\") pod \"maas-controller-5bd575b874-dd4ht\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") " pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:06.356084 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.355988    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-metrics-certs\") pod \"maas-controller-5bd575b874-dd4ht\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") " pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:06.356084 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.356035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhtc6\" (UniqueName: \"kubernetes.io/projected/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-kube-api-access-rhtc6\") pod \"maas-controller-5bd575b874-dd4ht\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") " pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:06.456929 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.456908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-webhook-cert\") pod \"maas-controller-5bd575b874-dd4ht\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") " pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:06.457028 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.456948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-metrics-certs\") pod \"maas-controller-5bd575b874-dd4ht\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") " pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:06.457028 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.456970    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhtc6\" (UniqueName: \"kubernetes.io/projected/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-kube-api-access-rhtc6\") pod \"maas-controller-5bd575b874-dd4ht\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") " pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:06.457178 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:53:06.457062    2569 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 18 13:53:06.457178 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:53:06.457172    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-webhook-cert podName:9ea6c357-f700-4d2b-9cab-28fe91f89cfa nodeName:}" failed. No retries permitted until 2026-09-18 13:53:06.957152621 +0000 UTC m=+528.859608801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-webhook-cert") pod "maas-controller-5bd575b874-dd4ht" (UID: "9ea6c357-f700-4d2b-9cab-28fe91f89cfa") : secret "maas-controller-webhook-cert" not found
Sep 18 13:53:06.459368 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.459348    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-metrics-certs\") pod \"maas-controller-5bd575b874-dd4ht\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") " pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:06.465660 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.465642    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhtc6\" (UniqueName: \"kubernetes.io/projected/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-kube-api-access-rhtc6\") pod \"maas-controller-5bd575b874-dd4ht\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") " pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:06.760471 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.760446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-webhook-cert\") pod \"maas-controller-57ddb66d95-skpmb\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") " pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.762731 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.762711    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-webhook-cert\") pod \"maas-controller-57ddb66d95-skpmb\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") " pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.939982 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.939959    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:06.962130 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.962111    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-webhook-cert\") pod \"maas-controller-5bd575b874-dd4ht\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") " pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:06.969926 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:06.964664    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-webhook-cert\") pod \"maas-controller-5bd575b874-dd4ht\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") " pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:07.152573 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:07.152512    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:07.213099 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:07.213053    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57ddb66d95-skpmb"]
Sep 18 13:53:07.215254 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:53:07.215228    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff86707_d908_4581_b818_604719733eb3.slice/crio-e6e30da405a631b43d931a5061b403fcb1734640f0ad30e1144b554dac4b3afd WatchSource:0}: Error finding container e6e30da405a631b43d931a5061b403fcb1734640f0ad30e1144b554dac4b3afd: Status 404 returned error can't find the container with id e6e30da405a631b43d931a5061b403fcb1734640f0ad30e1144b554dac4b3afd
Sep 18 13:53:07.421986 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:07.421959    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bd575b874-dd4ht"]
Sep 18 13:53:07.424409 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:53:07.424379    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea6c357_f700_4d2b_9cab_28fe91f89cfa.slice/crio-e8134c029a7bea16c28a157f2b0110fa1fd636d2a21638598fd5b9ddd19e9a04 WatchSource:0}: Error finding container e8134c029a7bea16c28a157f2b0110fa1fd636d2a21638598fd5b9ddd19e9a04: Status 404 returned error can't find the container with id e8134c029a7bea16c28a157f2b0110fa1fd636d2a21638598fd5b9ddd19e9a04
Sep 18 13:53:07.650611 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:07.650572    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bd575b874-dd4ht" event={"ID":"9ea6c357-f700-4d2b-9cab-28fe91f89cfa","Type":"ContainerStarted","Data":"e8134c029a7bea16c28a157f2b0110fa1fd636d2a21638598fd5b9ddd19e9a04"}
Sep 18 13:53:07.651889 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:07.651868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57ddb66d95-skpmb" event={"ID":"eff86707-d908-4581-b818-604719733eb3","Type":"ContainerStarted","Data":"e6e30da405a631b43d931a5061b403fcb1734640f0ad30e1144b554dac4b3afd"}
Sep 18 13:53:10.667544 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:10.667507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bd575b874-dd4ht" event={"ID":"9ea6c357-f700-4d2b-9cab-28fe91f89cfa","Type":"ContainerStarted","Data":"d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9"}
Sep 18 13:53:10.667972 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:10.667570    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:10.669093 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:10.669044    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57ddb66d95-skpmb" event={"ID":"eff86707-d908-4581-b818-604719733eb3","Type":"ContainerStarted","Data":"ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891"}
Sep 18 13:53:10.669225 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:10.669199    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:10.692296 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:10.692252    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bd575b874-dd4ht" podStartSLOduration=2.24981494 podStartE2EDuration="4.692238326s" totalImagesPullingTime="2.442423386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:53:10.690603311 +0000 UTC m=+532.593059496" watchObservedRunningTime="2026-09-18 13:53:10.692238326 +0000 UTC m=+532.594694509"
Sep 18 13:53:10.712488 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:10.712442    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57ddb66d95-skpmb" podStartSLOduration=2.051255317 podStartE2EDuration="4.712430962s" totalImagesPullingTime="2.661175645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:53:10.710852386 +0000 UTC m=+532.613308571" watchObservedRunningTime="2026-09-18 13:53:10.712430962 +0000 UTC m=+532.614887146"
Sep 18 13:53:21.676036 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:21.676005    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:21.676453 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:21.676179    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 13:53:21.739022 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:21.738990    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57ddb66d95-skpmb"]
Sep 18 13:53:21.739241 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:21.739180    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57ddb66d95-skpmb" podUID="eff86707-d908-4581-b818-604719733eb3" containerName="manager" containerID="cri-o://ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891" gracePeriod=30
Sep 18 13:53:21.973769 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:21.973748    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:22.076677 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.076651    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-metrics-certs\") pod \"eff86707-d908-4581-b818-604719733eb3\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") "
Sep 18 13:53:22.076854 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.076756    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2wxk\" (UniqueName: \"kubernetes.io/projected/eff86707-d908-4581-b818-604719733eb3-kube-api-access-m2wxk\") pod \"eff86707-d908-4581-b818-604719733eb3\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") "
Sep 18 13:53:22.076854 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.076797    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-webhook-cert\") pod \"eff86707-d908-4581-b818-604719733eb3\" (UID: \"eff86707-d908-4581-b818-604719733eb3\") "
Sep 18 13:53:22.078986 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.078956    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff86707-d908-4581-b818-604719733eb3-kube-api-access-m2wxk" (OuterVolumeSpecName: "kube-api-access-m2wxk") pod "eff86707-d908-4581-b818-604719733eb3" (UID: "eff86707-d908-4581-b818-604719733eb3"). InnerVolumeSpecName "kube-api-access-m2wxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:53:22.079112 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.079012    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "eff86707-d908-4581-b818-604719733eb3" (UID: "eff86707-d908-4581-b818-604719733eb3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:53:22.079112 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.079025    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eff86707-d908-4581-b818-604719733eb3" (UID: "eff86707-d908-4581-b818-604719733eb3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:53:22.177711 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.177674    2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-webhook-cert\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:53:22.177711 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.177706    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eff86707-d908-4581-b818-604719733eb3-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:53:22.177894 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.177719    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2wxk\" (UniqueName: \"kubernetes.io/projected/eff86707-d908-4581-b818-604719733eb3-kube-api-access-m2wxk\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:53:22.721370 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.721340    2569 generic.go:356] "Generic (PLEG): container finished" podID="eff86707-d908-4581-b818-604719733eb3" containerID="ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891" exitCode=0
Sep 18 13:53:22.721752 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.721383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57ddb66d95-skpmb" event={"ID":"eff86707-d908-4581-b818-604719733eb3","Type":"ContainerDied","Data":"ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891"}
Sep 18 13:53:22.721752 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.721400    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57ddb66d95-skpmb"
Sep 18 13:53:22.721752 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.721431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57ddb66d95-skpmb" event={"ID":"eff86707-d908-4581-b818-604719733eb3","Type":"ContainerDied","Data":"e6e30da405a631b43d931a5061b403fcb1734640f0ad30e1144b554dac4b3afd"}
Sep 18 13:53:22.721752 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.721464    2569 scope.go:117] "RemoveContainer" containerID="ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891"
Sep 18 13:53:22.729335 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.729316    2569 scope.go:117] "RemoveContainer" containerID="ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891"
Sep 18 13:53:22.729600 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:53:22.729583    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891\": container with ID starting with ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891 not found: ID does not exist" containerID="ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891"
Sep 18 13:53:22.729649 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.729607    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891"} err="failed to get container status \"ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891\": rpc error: code = NotFound desc = could not find container \"ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891\": container with ID starting with ef13ee6b59a703e600e43b5fb2bc03e2b9f6835e3a1f76c517a7f17ed40b6891 not found: ID does not exist"
Sep 18 13:53:22.739016 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.738994    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57ddb66d95-skpmb"]
Sep 18 13:53:22.740968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:22.740948    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57ddb66d95-skpmb"]
Sep 18 13:53:24.485109 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:24.485057    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff86707-d908-4581-b818-604719733eb3" path="/var/lib/kubelet/pods/eff86707-d908-4581-b818-604719733eb3/volumes"
Sep 18 13:53:43.209685 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.209654    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"]
Sep 18 13:53:43.210332 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.210082    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff86707-d908-4581-b818-604719733eb3" containerName="manager"
Sep 18 13:53:43.210332 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.210100    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff86707-d908-4581-b818-604719733eb3" containerName="manager"
Sep 18 13:53:43.210332 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.210199    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff86707-d908-4581-b818-604719733eb3" containerName="manager"
Sep 18 13:53:43.215606 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.215589    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.218275 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.218255    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 13:53:43.219543 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.219525    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vx9xd\""
Sep 18 13:53:43.219642 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.219594    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:53:43.219642 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.219526    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 13:53:43.219758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.219601    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 13:53:43.220776 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.220755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-metrics-certs\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.220932 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.220916    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b11d110-87d2-4e58-bf10-1f27550f0238-openshift-service-ca\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.221138 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.221120    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-maas-api-tls\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.221257 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.221240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9wgm\" (UniqueName: \"kubernetes.io/projected/5b11d110-87d2-4e58-bf10-1f27550f0238-kube-api-access-s9wgm\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.222101 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.222061    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"]
Sep 18 13:53:43.321670 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.321639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-metrics-certs\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.321788 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.321681    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b11d110-87d2-4e58-bf10-1f27550f0238-openshift-service-ca\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.321788 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.321733    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-maas-api-tls\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.321900 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:53:43.321801    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 18 13:53:43.321900 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:53:43.321872    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-metrics-certs podName:5b11d110-87d2-4e58-bf10-1f27550f0238 nodeName:}" failed. No retries permitted until 2026-09-18 13:53:43.82185245 +0000 UTC m=+565.724308619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-metrics-certs") pod "maas-api-859bfcbfc5-2nxnn" (UID: "5b11d110-87d2-4e58-bf10-1f27550f0238") : secret "maas-api-metrics-tls" not found
Sep 18 13:53:43.321900 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.321891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9wgm\" (UniqueName: \"kubernetes.io/projected/5b11d110-87d2-4e58-bf10-1f27550f0238-kube-api-access-s9wgm\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.322483 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.322466    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b11d110-87d2-4e58-bf10-1f27550f0238-openshift-service-ca\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.324401 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.324381    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-maas-api-tls\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.331113 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.331095    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9wgm\" (UniqueName: \"kubernetes.io/projected/5b11d110-87d2-4e58-bf10-1f27550f0238-kube-api-access-s9wgm\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.825283 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.825248    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-metrics-certs\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:43.827884 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:43.827855    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-metrics-certs\") pod \"maas-api-859bfcbfc5-2nxnn\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") " pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:44.127138 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:44.127041    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:44.408336 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:44.408314    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"]
Sep 18 13:53:44.411063 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:53:44.411027    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b11d110_87d2_4e58_bf10_1f27550f0238.slice/crio-6a1e427d0925181499f8318b45176521713a1ea3529e28c2ce020d6895835357 WatchSource:0}: Error finding container 6a1e427d0925181499f8318b45176521713a1ea3529e28c2ce020d6895835357: Status 404 returned error can't find the container with id 6a1e427d0925181499f8318b45176521713a1ea3529e28c2ce020d6895835357
Sep 18 13:53:44.813110 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:44.813061    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn" event={"ID":"5b11d110-87d2-4e58-bf10-1f27550f0238","Type":"ContainerStarted","Data":"6a1e427d0925181499f8318b45176521713a1ea3529e28c2ce020d6895835357"}
Sep 18 13:53:46.823830 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:46.823800    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn" event={"ID":"5b11d110-87d2-4e58-bf10-1f27550f0238","Type":"ContainerStarted","Data":"41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c"}
Sep 18 13:53:46.824197 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:46.823894    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:53:46.842580 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:46.842526    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn" podStartSLOduration=2.193494042 podStartE2EDuration="3.842511386s" totalImagesPullingTime="1.649017344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:53:46.842427845 +0000 UTC m=+568.744884031" watchObservedRunningTime="2026-09-18 13:53:46.842511386 +0000 UTC m=+568.744967571"
Sep 18 13:53:52.834173 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:53:52.834145    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 13:54:00.706493 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.706466    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"]
Sep 18 13:54:00.710313 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.710290    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.713407 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.713386    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 13:54:00.713536 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.713449    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 13:54:00.714395 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.714378    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 18 13:54:00.714512 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.714384    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mt56b\""
Sep 18 13:54:00.722115 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.722093    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"]
Sep 18 13:54:00.737475 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.737457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.737588 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.737517    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.737633 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.737604    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35583c22-c074-4bf7-bdeb-6af8abb2f354-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.737633 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.737624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.737710 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.737642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.737710 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.737658    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whp8b\" (UniqueName: \"kubernetes.io/projected/35583c22-c074-4bf7-bdeb-6af8abb2f354-kube-api-access-whp8b\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.838379 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.838351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.838500 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.838422    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35583c22-c074-4bf7-bdeb-6af8abb2f354-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.838500 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.838456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.838500 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.838482    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.838661 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.838504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whp8b\" (UniqueName: \"kubernetes.io/projected/35583c22-c074-4bf7-bdeb-6af8abb2f354-kube-api-access-whp8b\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.838661 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.838540    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.838773 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.838753    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.838837 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.838775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.838923 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.838907    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.840812 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.840792    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35583c22-c074-4bf7-bdeb-6af8abb2f354-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.841020 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.841001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35583c22-c074-4bf7-bdeb-6af8abb2f354-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:00.848704 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:00.848684    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whp8b\" (UniqueName: \"kubernetes.io/projected/35583c22-c074-4bf7-bdeb-6af8abb2f354-kube-api-access-whp8b\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs\" (UID: \"35583c22-c074-4bf7-bdeb-6af8abb2f354\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:01.020773 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:01.020698    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:01.292601 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:01.292578    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"]
Sep 18 13:54:01.295155 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:54:01.295129    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35583c22_c074_4bf7_bdeb_6af8abb2f354.slice/crio-b17bbc8b41f7b0b325283ef1500133de39c2c0becbda1efde8a55e667c1ddce5 WatchSource:0}: Error finding container b17bbc8b41f7b0b325283ef1500133de39c2c0becbda1efde8a55e667c1ddce5: Status 404 returned error can't find the container with id b17bbc8b41f7b0b325283ef1500133de39c2c0becbda1efde8a55e667c1ddce5
Sep 18 13:54:01.888592 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:01.888557    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs" event={"ID":"35583c22-c074-4bf7-bdeb-6af8abb2f354","Type":"ContainerStarted","Data":"b17bbc8b41f7b0b325283ef1500133de39c2c0becbda1efde8a55e667c1ddce5"}
Sep 18 13:54:02.893690 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:02.893658    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs" event={"ID":"35583c22-c074-4bf7-bdeb-6af8abb2f354","Type":"ContainerStarted","Data":"967363a1b9900f048bcc7ec90ecead81ac76aedb3a052e0a2e37dca8eaeac7a3"}
Sep 18 13:54:02.913863 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:02.913820    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs" podStartSLOduration=1.835577585 podStartE2EDuration="2.913804457s" totalImagesPullingTime="1.078226872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:54:02.912625496 +0000 UTC m=+584.815081671" watchObservedRunningTime="2026-09-18 13:54:02.913804457 +0000 UTC m=+584.816260639"
Sep 18 13:54:10.495197 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.495164    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"]
Sep 18 13:54:10.498683 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.498663    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.501502 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.501482    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 18 13:54:10.508877 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.508859    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"]
Sep 18 13:54:10.608015 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.607989    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.608190 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.608022    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm9zc\" (UniqueName: \"kubernetes.io/projected/bfaf8547-6873-4ae9-b9a7-cbfded37c070-kube-api-access-xm9zc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.608190 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.608047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfaf8547-6873-4ae9-b9a7-cbfded37c070-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.608190 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.608109    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.608190 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.608161    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.608190 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.608176    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.708532 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.708496    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.708701 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.708543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm9zc\" (UniqueName: \"kubernetes.io/projected/bfaf8547-6873-4ae9-b9a7-cbfded37c070-kube-api-access-xm9zc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.708701 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.708577    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfaf8547-6873-4ae9-b9a7-cbfded37c070-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.708701 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.708600    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.708701 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.708631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.708916 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.708765    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.708916 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.708886    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.709023 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.708993    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.709116 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.709097    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.710974 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.710955    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfaf8547-6873-4ae9-b9a7-cbfded37c070-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.711275 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.711260    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfaf8547-6873-4ae9-b9a7-cbfded37c070-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.718022 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.718002    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm9zc\" (UniqueName: \"kubernetes.io/projected/bfaf8547-6873-4ae9-b9a7-cbfded37c070-kube-api-access-xm9zc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g2bjk\" (UID: \"bfaf8547-6873-4ae9-b9a7-cbfded37c070\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:10.809030 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:10.808972    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:11.021648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:11.021623    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:11.021795 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:11.021677    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:11.033725 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:11.033705    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:11.078349 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:11.078328    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"]
Sep 18 13:54:11.080331 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:54:11.080307    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaf8547_6873_4ae9_b9a7_cbfded37c070.slice/crio-7c820e719b242f77cb1c7c99038e4e4e2822470c0de510f88998f57d242d09ce WatchSource:0}: Error finding container 7c820e719b242f77cb1c7c99038e4e4e2822470c0de510f88998f57d242d09ce: Status 404 returned error can't find the container with id 7c820e719b242f77cb1c7c99038e4e4e2822470c0de510f88998f57d242d09ce
Sep 18 13:54:11.932544 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:11.932511    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk" event={"ID":"bfaf8547-6873-4ae9-b9a7-cbfded37c070","Type":"ContainerStarted","Data":"2e7c85586a9a5e99a0997380478c9daa2542aea08ecfc31bbf5c2074d0d3ccd7"}
Sep 18 13:54:11.932937 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:11.932561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk" event={"ID":"bfaf8547-6873-4ae9-b9a7-cbfded37c070","Type":"ContainerStarted","Data":"7c820e719b242f77cb1c7c99038e4e4e2822470c0de510f88998f57d242d09ce"}
Sep 18 13:54:11.943173 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:11.943153    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs"
Sep 18 13:54:11.955305 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:11.955265    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk" podStartSLOduration=1.769177892 podStartE2EDuration="1.955251841s" totalImagesPullingTime="186.073949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:54:11.953150695 +0000 UTC m=+593.855606880" watchObservedRunningTime="2026-09-18 13:54:11.955251841 +0000 UTC m=+593.857708025"
Sep 18 13:54:18.458637 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:18.458612    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:54:18.464767 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:18.464749    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:54:20.809154 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:20.809123    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:20.809570 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:20.809188    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:20.820947 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:20.820920    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:54:20.980465 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:54:20.980443    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g2bjk"
Sep 18 13:56:10.829135 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.828884    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"]
Sep 18 13:56:10.832628 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.832605    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:10.835487 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.835419    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ed4c53-openshift-default-dockercfg-h6nj8\""
Sep 18 13:56:10.845402 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.845381    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"]
Sep 18 13:56:10.968526 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.968495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcpc4\" (UniqueName: \"kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-kube-api-access-mcpc4\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:10.968526 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.968530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-socket\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:10.968717 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.968548    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-envoy\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:10.968717 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.968619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istiod-ca-cert\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:10.968717 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.968649    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-token\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:10.968717 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.968675    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-data\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:10.968717 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.968711    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-credential-socket\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:10.968883 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.968732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-certs\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:10.968883 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:10.968750    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-podinfo\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.069710 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.069679    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-certs\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.069894 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.069721    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-podinfo\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.069894 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.069773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcpc4\" (UniqueName: \"kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-kube-api-access-mcpc4\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.069894 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.069799    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-socket\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.069894 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.069823    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-envoy\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.070162 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.069957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istiod-ca-cert\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.070162 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.070012    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-token\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.070162 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.070060    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-data\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.070162 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.070132    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-credential-socket\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.070162 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.070141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-certs\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.070505 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.070483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-socket\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.070572 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.070541    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-credential-socket\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.070626 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.070605    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-data\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.070754 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.070734    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istiod-ca-cert\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.072553 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.072534    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-envoy\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.073182 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.073145    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-podinfo\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.081264 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.081217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-token\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.081376 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.081358    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcpc4\" (UniqueName: \"kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-kube-api-access-mcpc4\") pod \"e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") " pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.146027 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.146001    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:11.422355 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.422333    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"]
Sep 18 13:56:11.424084 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:56:11.424050    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0241500c_e3f7_43dc_895e_5e3ff4bc0275.slice/crio-b6060390eeacdff0c07c0dd0762c936b4013768023fddba58b44db36568be45d WatchSource:0}: Error finding container b6060390eeacdff0c07c0dd0762c936b4013768023fddba58b44db36568be45d: Status 404 returned error can't find the container with id b6060390eeacdff0c07c0dd0762c936b4013768023fddba58b44db36568be45d
Sep 18 13:56:11.425937 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.425922    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:56:11.451612 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:11.451582    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" event={"ID":"0241500c-e3f7-43dc-895e-5e3ff4bc0275","Type":"ContainerStarted","Data":"b6060390eeacdff0c07c0dd0762c936b4013768023fddba58b44db36568be45d"}
Sep 18 13:56:15.109137 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.109103    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"]
Sep 18 13:56:15.114918 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.114893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.118046 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.117871    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-df2eb8-serving-cert\""
Sep 18 13:56:15.127870 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.127809    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"]
Sep 18 13:56:15.203276 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.203247    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-maas-api-tls\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.203430 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.203302    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e953bc-4256-4fd6-88d0-244131c1ccc6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.203430 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.203384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-metrics-certs\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.203539 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.203448    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcs6p\" (UniqueName: \"kubernetes.io/projected/e5e953bc-4256-4fd6-88d0-244131c1ccc6-kube-api-access-kcs6p\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.307063 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.306700    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-metrics-certs\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.307063 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.306751    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcs6p\" (UniqueName: \"kubernetes.io/projected/e5e953bc-4256-4fd6-88d0-244131c1ccc6-kube-api-access-kcs6p\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.307063 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.306831    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-maas-api-tls\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.307063 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.306870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e953bc-4256-4fd6-88d0-244131c1ccc6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.307986 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.307665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e953bc-4256-4fd6-88d0-244131c1ccc6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.310780 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.310756    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-metrics-certs\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.311246 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.311225    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-maas-api-tls\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.316697 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.316675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcs6p\" (UniqueName: \"kubernetes.io/projected/e5e953bc-4256-4fd6-88d0-244131c1ccc6-kube-api-access-kcs6p\") pod \"maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.429712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.429645    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:15.711094 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:15.711041    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"]
Sep 18 13:56:15.715310 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:56:15.715280    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e953bc_4256_4fd6_88d0_244131c1ccc6.slice/crio-3b41d8e60fef0318eaab3fb7a6ef447719f0e8395ef251281c322faf84247d74 WatchSource:0}: Error finding container 3b41d8e60fef0318eaab3fb7a6ef447719f0e8395ef251281c322faf84247d74: Status 404 returned error can't find the container with id 3b41d8e60fef0318eaab3fb7a6ef447719f0e8395ef251281c322faf84247d74
Sep 18 13:56:16.186848 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.186766    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"]
Sep 18 13:56:16.191708 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.191673    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.194363 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.194240    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-377ca2-serving-cert\""
Sep 18 13:56:16.201435 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.201410    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"]
Sep 18 13:56:16.317609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.317575    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-metrics-certs\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.317787 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.317679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.317787 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.317704    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz7xm\" (UniqueName: \"kubernetes.io/projected/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-kube-api-access-tz7xm\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.317787 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.317732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.418435 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.418405    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.418435 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.418440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz7xm\" (UniqueName: \"kubernetes.io/projected/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-kube-api-access-tz7xm\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.418686 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.418460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.418686 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.418513    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-metrics-certs\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.418686 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:56:16.418573    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-serving-cert: secret "maas-api-e2e-worker-w4-377ca2-serving-cert" not found
Sep 18 13:56:16.418686 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:56:16.418643    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-maas-api-tls podName:ed1c0dd6-c519-4bc5-8b09-c7d30399073d nodeName:}" failed. No retries permitted until 2026-09-18 13:56:16.918622697 +0000 UTC m=+718.821078874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-maas-api-tls") pod "maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g" (UID: "ed1c0dd6-c519-4bc5-8b09-c7d30399073d") : secret "maas-api-e2e-worker-w4-377ca2-serving-cert" not found
Sep 18 13:56:16.419250 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.419224    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.421030 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.421005    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-metrics-certs\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.430733 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.430706    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz7xm\" (UniqueName: \"kubernetes.io/projected/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-kube-api-access-tz7xm\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.476826 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.476795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp" event={"ID":"e5e953bc-4256-4fd6-88d0-244131c1ccc6","Type":"ContainerStarted","Data":"3b41d8e60fef0318eaab3fb7a6ef447719f0e8395ef251281c322faf84247d74"}
Sep 18 13:56:16.855399 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.855334    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"]
Sep 18 13:56:16.862474 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.862454    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 13:56:16.866318 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.866259    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-377ca2\""
Sep 18 13:56:16.867628 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.867606    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-377ca2-dockercfg-68xdt\""
Sep 18 13:56:16.876031 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.876007    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"]
Sep 18 13:56:16.924283 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.924243    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:16.927027 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:16.927001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:17.025523 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.025492    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntmg\" (UniqueName: \"kubernetes.io/projected/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-kube-api-access-gntmg\") pod \"payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg\" (UID: \"2848a5b6-e7ed-4fbc-a750-556cfea2a76a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 13:56:17.025679 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.025619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg\" (UID: \"2848a5b6-e7ed-4fbc-a750-556cfea2a76a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 13:56:17.104449 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.104418    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:17.126829 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.126797    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntmg\" (UniqueName: \"kubernetes.io/projected/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-kube-api-access-gntmg\") pod \"payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg\" (UID: \"2848a5b6-e7ed-4fbc-a750-556cfea2a76a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 13:56:17.126951 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.126888    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg\" (UID: \"2848a5b6-e7ed-4fbc-a750-556cfea2a76a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 13:56:17.127663 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.127646    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg\" (UID: \"2848a5b6-e7ed-4fbc-a750-556cfea2a76a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 13:56:17.140242 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.140216    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntmg\" (UniqueName: \"kubernetes.io/projected/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-kube-api-access-gntmg\") pod \"payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg\" (UID: \"2848a5b6-e7ed-4fbc-a750-556cfea2a76a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 13:56:17.174055 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.174027    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 13:56:17.496501 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.496475    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:56:17.496817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.496542    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:56:17.496817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.496576    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:56:17.684538 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.684511    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"]
Sep 18 13:56:17.687424 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:17.687353    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"]
Sep 18 13:56:17.687424 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:56:17.687353    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1c0dd6_c519_4bc5_8b09_c7d30399073d.slice/crio-db07b56daa272abe0aea9edc5921f0795dcf5d3fe3f9dc3386373a0a17df2e7f WatchSource:0}: Error finding container db07b56daa272abe0aea9edc5921f0795dcf5d3fe3f9dc3386373a0a17df2e7f: Status 404 returned error can't find the container with id db07b56daa272abe0aea9edc5921f0795dcf5d3fe3f9dc3386373a0a17df2e7f
Sep 18 13:56:17.689581 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:56:17.689460    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2848a5b6_e7ed_4fbc_a750_556cfea2a76a.slice/crio-0015622219fceeb1d2211c73407b82662e8094509d041ac199a65414db9b462b WatchSource:0}: Error finding container 0015622219fceeb1d2211c73407b82662e8094509d041ac199a65414db9b462b: Status 404 returned error can't find the container with id 0015622219fceeb1d2211c73407b82662e8094509d041ac199a65414db9b462b
Sep 18 13:56:18.490814 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:18.490774    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" event={"ID":"0241500c-e3f7-43dc-895e-5e3ff4bc0275","Type":"ContainerStarted","Data":"3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3"}
Sep 18 13:56:18.494901 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:18.494870    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g" event={"ID":"ed1c0dd6-c519-4bc5-8b09-c7d30399073d","Type":"ContainerStarted","Data":"e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99"}
Sep 18 13:56:18.495081 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:18.494924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g" event={"ID":"ed1c0dd6-c519-4bc5-8b09-c7d30399073d","Type":"ContainerStarted","Data":"db07b56daa272abe0aea9edc5921f0795dcf5d3fe3f9dc3386373a0a17df2e7f"}
Sep 18 13:56:18.495613 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:18.495589    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:18.497218 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:18.497193    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg" event={"ID":"2848a5b6-e7ed-4fbc-a750-556cfea2a76a","Type":"ContainerStarted","Data":"0015622219fceeb1d2211c73407b82662e8094509d041ac199a65414db9b462b"}
Sep 18 13:56:18.499361 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:18.499314    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp" event={"ID":"e5e953bc-4256-4fd6-88d0-244131c1ccc6","Type":"ContainerStarted","Data":"96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2"}
Sep 18 13:56:18.499524 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:18.499501    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:18.516648 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:18.516270    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" podStartSLOduration=2.4461287 podStartE2EDuration="8.516253712s" totalImagesPullingTime="6.070125012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:18.512637173 +0000 UTC m=+720.415093357" watchObservedRunningTime="2026-09-18 13:56:18.516253712 +0000 UTC m=+720.418709896"
Sep 18 13:56:18.536244 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:18.536205    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp" podStartSLOduration=1.757395393 podStartE2EDuration="3.536188826s" totalImagesPullingTime="1.778793433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:18.533867936 +0000 UTC m=+720.436324123" watchObservedRunningTime="2026-09-18 13:56:18.536188826 +0000 UTC m=+720.438645011"
Sep 18 13:56:18.559229 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:18.559096    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g" podStartSLOduration=2.241052533 podStartE2EDuration="2.559046847s" totalImagesPullingTime="317.994314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:18.556111507 +0000 UTC m=+720.458567694" watchObservedRunningTime="2026-09-18 13:56:18.559046847 +0000 UTC m=+720.461503040"
Sep 18 13:56:19.146218 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:19.146186    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:19.160510 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:19.159917    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:19.505799 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:19.505766    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:19.507194 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:19.507168    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:20.511815 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:20.511776    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg" event={"ID":"2848a5b6-e7ed-4fbc-a750-556cfea2a76a","Type":"ContainerStarted","Data":"708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845"}
Sep 18 13:56:20.512317 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:20.512298    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 13:56:20.559704 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:20.559657    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg" podStartSLOduration=2.185274562 podStartE2EDuration="4.559643546s" totalImagesPullingTime="2.374368984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:20.547642581 +0000 UTC m=+722.450098788" watchObservedRunningTime="2026-09-18 13:56:20.559643546 +0000 UTC m=+722.462099794"
Sep 18 13:56:22.728141 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.728107    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"]
Sep 18 13:56:22.732009 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.731987    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.735337 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.735314    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ed4c53-serving-cert\""
Sep 18 13:56:22.748750 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.748727    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"]
Sep 18 13:56:22.878977 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.878950    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czbq\" (UniqueName: \"kubernetes.io/projected/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-kube-api-access-4czbq\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.878977 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.878981    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.879242 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.879024    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.879242 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.879042    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-metrics-certs\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.981359 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.981285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czbq\" (UniqueName: \"kubernetes.io/projected/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-kube-api-access-4czbq\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.981507 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.981362    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.981507 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.981431    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.981507 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.981450    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-metrics-certs\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.982212 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.982188    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.984083 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.984034    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-metrics-certs\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.984170 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.984041    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:22.995099 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:22.995057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czbq\" (UniqueName: \"kubernetes.io/projected/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-kube-api-access-4czbq\") pod \"maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:23.042154 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:23.042129    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:23.313542 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:23.313520    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"]
Sep 18 13:56:23.315212 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:56:23.315179    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349ba1a7_c10a_43f5_b853_ca6fb6872ba8.slice/crio-5268db1ba4685b4d2c22f35d9a3b81df31e67e6a6f2b8f45d851c7caf4928d47 WatchSource:0}: Error finding container 5268db1ba4685b4d2c22f35d9a3b81df31e67e6a6f2b8f45d851c7caf4928d47: Status 404 returned error can't find the container with id 5268db1ba4685b4d2c22f35d9a3b81df31e67e6a6f2b8f45d851c7caf4928d47
Sep 18 13:56:23.526265 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:23.526227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v" event={"ID":"349ba1a7-c10a-43f5-b853-ca6fb6872ba8","Type":"ContainerStarted","Data":"5268db1ba4685b4d2c22f35d9a3b81df31e67e6a6f2b8f45d851c7caf4928d47"}
Sep 18 13:56:24.511005 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:24.510975    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 13:56:24.533305 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:24.533282    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v" event={"ID":"349ba1a7-c10a-43f5-b853-ca6fb6872ba8","Type":"ContainerStarted","Data":"2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11"}
Sep 18 13:56:24.533461 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:24.533400    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:24.557745 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:24.557700    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v" podStartSLOduration=2.152168139 podStartE2EDuration="2.557685901s" totalImagesPullingTime="405.517762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:24.55640139 +0000 UTC m=+726.458857575" watchObservedRunningTime="2026-09-18 13:56:24.557685901 +0000 UTC m=+726.460142101"
Sep 18 13:56:25.517358 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:25.517333    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 13:56:27.061466 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.061428    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"]
Sep 18 13:56:27.070735 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.070711    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.076353 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.076326    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-70f06e-serving-cert\""
Sep 18 13:56:27.076629 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.076605    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"]
Sep 18 13:56:27.220465 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.220433    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-metrics-certs\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.220650 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.220576    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/859f21d1-3103-44ee-998a-7c0c8fbcac32-openshift-service-ca\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.220650 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.220602    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-maas-api-tls\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.220650 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.220619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxnw\" (UniqueName: \"kubernetes.io/projected/859f21d1-3103-44ee-998a-7c0c8fbcac32-kube-api-access-6wxnw\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.321714 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.321653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/859f21d1-3103-44ee-998a-7c0c8fbcac32-openshift-service-ca\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.321714 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.321682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-maas-api-tls\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.321714 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.321699    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxnw\" (UniqueName: \"kubernetes.io/projected/859f21d1-3103-44ee-998a-7c0c8fbcac32-kube-api-access-6wxnw\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.321930 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.321779    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-metrics-certs\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.321930 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:56:27.321830    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-serving-cert: secret "maas-api-e2e-autores-70f06e-serving-cert" not found
Sep 18 13:56:27.321930 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:56:27.321904    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-maas-api-tls podName:859f21d1-3103-44ee-998a-7c0c8fbcac32 nodeName:}" failed. No retries permitted until 2026-09-18 13:56:27.821883911 +0000 UTC m=+729.724340074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-maas-api-tls") pod "maas-api-e2e-autores-70f06e-6cd57685f5-sklxt" (UID: "859f21d1-3103-44ee-998a-7c0c8fbcac32") : secret "maas-api-e2e-autores-70f06e-serving-cert" not found
Sep 18 13:56:27.322314 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.322294    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/859f21d1-3103-44ee-998a-7c0c8fbcac32-openshift-service-ca\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.324423 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.324405    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-metrics-certs\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.334365 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.334340    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxnw\" (UniqueName: \"kubernetes.io/projected/859f21d1-3103-44ee-998a-7c0c8fbcac32-kube-api-access-6wxnw\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.827695 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.827657    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-maas-api-tls\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.831198 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.831171    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-maas-api-tls\") pod \"maas-api-e2e-autores-70f06e-6cd57685f5-sklxt\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:27.983625 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:27.983598    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:28.258731 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:28.258706    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"]
Sep 18 13:56:28.259637 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:56:28.259614    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859f21d1_3103_44ee_998a_7c0c8fbcac32.slice/crio-3bda572d6afe70f30670faaf03f196ecf0b520046654af5021ce5f8299a45d96 WatchSource:0}: Error finding container 3bda572d6afe70f30670faaf03f196ecf0b520046654af5021ce5f8299a45d96: Status 404 returned error can't find the container with id 3bda572d6afe70f30670faaf03f196ecf0b520046654af5021ce5f8299a45d96
Sep 18 13:56:28.555326 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:28.555247    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt" event={"ID":"859f21d1-3103-44ee-998a-7c0c8fbcac32","Type":"ContainerStarted","Data":"3bda572d6afe70f30670faaf03f196ecf0b520046654af5021ce5f8299a45d96"}
Sep 18 13:56:29.564087 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:29.564035    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt" event={"ID":"859f21d1-3103-44ee-998a-7c0c8fbcac32","Type":"ContainerStarted","Data":"75954e6e330ec8bbcc0c5664a23e7acbc87192bce9ab0f6ac17ee352e9e88b55"}
Sep 18 13:56:29.564474 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:29.564186    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:29.592736 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:29.592678    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt" podStartSLOduration=2.204075191 podStartE2EDuration="2.592661024s" totalImagesPullingTime="388.585833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:29.59124827 +0000 UTC m=+731.493704455" watchObservedRunningTime="2026-09-18 13:56:29.592661024 +0000 UTC m=+731.495117210"
Sep 18 13:56:30.545587 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:30.545562    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 13:56:32.522316 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:32.522280    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 13:56:34.208344 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.208314    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"]
Sep 18 13:56:34.212118 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.212095    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.215534 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.215511    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b63f26-serving-cert\""
Sep 18 13:56:34.229889 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.229863    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"]
Sep 18 13:56:34.282782 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.282755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnb4n\" (UniqueName: \"kubernetes.io/projected/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-kube-api-access-cnb4n\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.282917 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.282826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-maas-api-tls\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.282917 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.282864    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.282917 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.282910    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-metrics-certs\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.383863 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.383830    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnb4n\" (UniqueName: \"kubernetes.io/projected/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-kube-api-access-cnb4n\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.384026 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.383881    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-maas-api-tls\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.384026 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.383910    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.384026 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.383943    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-metrics-certs\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.384179 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:56:34.384029    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-serving-cert: secret "maas-api-e2e-inf-b-b63f26-serving-cert" not found
Sep 18 13:56:34.384179 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:56:34.384105    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-maas-api-tls podName:2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79 nodeName:}" failed. No retries permitted until 2026-09-18 13:56:34.884086992 +0000 UTC m=+736.786543159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-maas-api-tls") pod "maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9" (UID: "2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79") : secret "maas-api-e2e-inf-b-b63f26-serving-cert" not found
Sep 18 13:56:34.384678 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.384661    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.386487 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.386473    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-metrics-certs\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.396205 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.396183    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnb4n\" (UniqueName: \"kubernetes.io/projected/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-kube-api-access-cnb4n\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.888712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.888678    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-maas-api-tls\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:34.891217 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:34.891189    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-maas-api-tls\") pod \"maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:35.125867 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:35.125839    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:35.402314 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:35.402289    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"]
Sep 18 13:56:35.403775 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:56:35.403745    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f811cf4_a8ed_4d17_ab69_3bdc1eeb7d79.slice/crio-8d6b451e9c6c8017c25f8f715061f55fabeb376d1b1cb41dbfe47031f251293c WatchSource:0}: Error finding container 8d6b451e9c6c8017c25f8f715061f55fabeb376d1b1cb41dbfe47031f251293c: Status 404 returned error can't find the container with id 8d6b451e9c6c8017c25f8f715061f55fabeb376d1b1cb41dbfe47031f251293c
Sep 18 13:56:35.575184 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:35.575151    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:56:35.594298 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:35.594268    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9" event={"ID":"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79","Type":"ContainerStarted","Data":"8d6b451e9c6c8017c25f8f715061f55fabeb376d1b1cb41dbfe47031f251293c"}
Sep 18 13:56:36.600660 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:36.600631    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9" event={"ID":"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79","Type":"ContainerStarted","Data":"ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65"}
Sep 18 13:56:36.601059 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:36.600688    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:36.624579 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:36.624522    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9" podStartSLOduration=2.242729233 podStartE2EDuration="2.624502669s" totalImagesPullingTime="381.773436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:36.622525336 +0000 UTC m=+738.524981521" watchObservedRunningTime="2026-09-18 13:56:36.624502669 +0000 UTC m=+738.526958855"
Sep 18 13:56:42.611638 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:42.611611    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 13:56:46.650202 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:46.650170    2569 generic.go:356] "Generic (PLEG): container finished" podID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerID="3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3" exitCode=0
Sep 18 13:56:46.650564 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:46.650247    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" event={"ID":"0241500c-e3f7-43dc-895e-5e3ff4bc0275","Type":"ContainerDied","Data":"3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3"}
Sep 18 13:56:46.650658 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:46.650635    2569 scope.go:117] "RemoveContainer" containerID="3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3"
Sep 18 13:56:46.651773 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:46.651747    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:56:46.651842 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:46.651812    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:56:46.651879 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:46.651843    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:56:47.660713 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:47.660668    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" event={"ID":"0241500c-e3f7-43dc-895e-5e3ff4bc0275","Type":"ContainerStarted","Data":"f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e"}
Sep 18 13:56:48.146415 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:48.146327    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:48.147910 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:48.147871    2569 patch_prober.go:28] interesting pod/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 18 13:56:48.148049 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:48.147935    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 18 13:56:49.147164 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:49.147121    2569 patch_prober.go:28] interesting pod/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 18 13:56:49.147663 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:49.147195    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 18 13:56:49.506489 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:49.506451    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:50.147236 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:50.147186    2569 patch_prober.go:28] interesting pod/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 18 13:56:50.147719 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:50.147273    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 18 13:56:51.146610 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:51.146574    2569 patch_prober.go:28] interesting pod/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 18 13:56:51.146895 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:51.146866    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 18 13:56:52.147352 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:52.147316    2569 patch_prober.go:28] interesting pod/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 18 13:56:52.147935 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:52.147903    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 18 13:56:53.151466 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:53.151431    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:56:53.152320 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:56:53.152303    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 13:57:01.876320 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:01.876286    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"]
Sep 18 13:57:01.883553 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:01.883530    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:01.887591 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:01.887565    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3675964e-dockercfg-4bzzc\""
Sep 18 13:57:01.887721 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:01.887600    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3675964e\""
Sep 18 13:57:01.889258 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:01.889231    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"]
Sep 18 13:57:02.052930 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.052892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359f8ffd-44ac-46ce-a416-75d25a893b27-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw\" (UID: \"359f8ffd-44ac-46ce-a416-75d25a893b27\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:02.053140 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.052960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wpf\" (UniqueName: \"kubernetes.io/projected/359f8ffd-44ac-46ce-a416-75d25a893b27-kube-api-access-m4wpf\") pod \"payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw\" (UID: \"359f8ffd-44ac-46ce-a416-75d25a893b27\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:02.153892 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.153769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359f8ffd-44ac-46ce-a416-75d25a893b27-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw\" (UID: \"359f8ffd-44ac-46ce-a416-75d25a893b27\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:02.153892 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.153844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wpf\" (UniqueName: \"kubernetes.io/projected/359f8ffd-44ac-46ce-a416-75d25a893b27-kube-api-access-m4wpf\") pod \"payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw\" (UID: \"359f8ffd-44ac-46ce-a416-75d25a893b27\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:02.154489 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.154467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359f8ffd-44ac-46ce-a416-75d25a893b27-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw\" (UID: \"359f8ffd-44ac-46ce-a416-75d25a893b27\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:02.166270 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.166243    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wpf\" (UniqueName: \"kubernetes.io/projected/359f8ffd-44ac-46ce-a416-75d25a893b27-kube-api-access-m4wpf\") pod \"payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw\" (UID: \"359f8ffd-44ac-46ce-a416-75d25a893b27\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:02.195786 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.195760    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:02.476450 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.476358    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"]
Sep 18 13:57:02.478582 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:57:02.478552    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359f8ffd_44ac_46ce_a416_75d25a893b27.slice/crio-49b1c65f9429fdd96eb6ec18db105158094c119ccd5b46a1f3ef9788c45d832d WatchSource:0}: Error finding container 49b1c65f9429fdd96eb6ec18db105158094c119ccd5b46a1f3ef9788c45d832d: Status 404 returned error can't find the container with id 49b1c65f9429fdd96eb6ec18db105158094c119ccd5b46a1f3ef9788c45d832d
Sep 18 13:57:02.736737 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.736647    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw" event={"ID":"359f8ffd-44ac-46ce-a416-75d25a893b27","Type":"ContainerStarted","Data":"76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3"}
Sep 18 13:57:02.736737 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.736708    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw" event={"ID":"359f8ffd-44ac-46ce-a416-75d25a893b27","Type":"ContainerStarted","Data":"49b1c65f9429fdd96eb6ec18db105158094c119ccd5b46a1f3ef9788c45d832d"}
Sep 18 13:57:02.736908 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.736738    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:02.756107 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:02.756044    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw" podStartSLOduration=1.756029065 podStartE2EDuration="1.756029065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:57:02.754629805 +0000 UTC m=+764.657085991" watchObservedRunningTime="2026-09-18 13:57:02.756029065 +0000 UTC m=+764.658485249"
Sep 18 13:57:13.743317 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:13.743289    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:29.557879 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.557837    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"]
Sep 18 13:57:29.558845 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.558102    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw" podUID="359f8ffd-44ac-46ce-a416-75d25a893b27" containerName="payload-pre-processing" containerID="cri-o://76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3" gracePeriod=30
Sep 18 13:57:29.813313 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.813245    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:29.870884 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.870854    2569 generic.go:356] "Generic (PLEG): container finished" podID="359f8ffd-44ac-46ce-a416-75d25a893b27" containerID="76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3" exitCode=0
Sep 18 13:57:29.871057 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.870901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw" event={"ID":"359f8ffd-44ac-46ce-a416-75d25a893b27","Type":"ContainerDied","Data":"76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3"}
Sep 18 13:57:29.871057 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.870922    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"
Sep 18 13:57:29.871057 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.870950    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw" event={"ID":"359f8ffd-44ac-46ce-a416-75d25a893b27","Type":"ContainerDied","Data":"49b1c65f9429fdd96eb6ec18db105158094c119ccd5b46a1f3ef9788c45d832d"}
Sep 18 13:57:29.871057 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.870979    2569 scope.go:117] "RemoveContainer" containerID="76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3"
Sep 18 13:57:29.884275 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.884253    2569 scope.go:117] "RemoveContainer" containerID="76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3"
Sep 18 13:57:29.884602 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:57:29.884580    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3\": container with ID starting with 76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3 not found: ID does not exist" containerID="76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3"
Sep 18 13:57:29.884716 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.884606    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3"} err="failed to get container status \"76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3\": rpc error: code = NotFound desc = could not find container \"76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3\": container with ID starting with 76eb49190179c284ed222fa742435ca826ea5e5fb468021ec3eea93071c1f4f3 not found: ID does not exist"
Sep 18 13:57:29.898157 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.898139    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359f8ffd-44ac-46ce-a416-75d25a893b27-plugins-config-volume\") pod \"359f8ffd-44ac-46ce-a416-75d25a893b27\" (UID: \"359f8ffd-44ac-46ce-a416-75d25a893b27\") "
Sep 18 13:57:29.898256 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.898174    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4wpf\" (UniqueName: \"kubernetes.io/projected/359f8ffd-44ac-46ce-a416-75d25a893b27-kube-api-access-m4wpf\") pod \"359f8ffd-44ac-46ce-a416-75d25a893b27\" (UID: \"359f8ffd-44ac-46ce-a416-75d25a893b27\") "
Sep 18 13:57:29.898549 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.898527    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359f8ffd-44ac-46ce-a416-75d25a893b27-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "359f8ffd-44ac-46ce-a416-75d25a893b27" (UID: "359f8ffd-44ac-46ce-a416-75d25a893b27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:57:29.900410 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.900390    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359f8ffd-44ac-46ce-a416-75d25a893b27-kube-api-access-m4wpf" (OuterVolumeSpecName: "kube-api-access-m4wpf") pod "359f8ffd-44ac-46ce-a416-75d25a893b27" (UID: "359f8ffd-44ac-46ce-a416-75d25a893b27"). InnerVolumeSpecName "kube-api-access-m4wpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:57:29.999295 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.999265    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/359f8ffd-44ac-46ce-a416-75d25a893b27-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:57:29.999295 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:29.999291    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4wpf\" (UniqueName: \"kubernetes.io/projected/359f8ffd-44ac-46ce-a416-75d25a893b27-kube-api-access-m4wpf\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:57:30.195934 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:30.195906    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"]
Sep 18 13:57:30.201257 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:30.201236    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3675964e-79c857d598-glnkw"]
Sep 18 13:57:30.485560 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:30.485531    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="359f8ffd-44ac-46ce-a416-75d25a893b27" path="/var/lib/kubelet/pods/359f8ffd-44ac-46ce-a416-75d25a893b27/volumes"
Sep 18 13:57:34.800450 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.800412    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"]
Sep 18 13:57:34.800968 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.800954    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359f8ffd-44ac-46ce-a416-75d25a893b27" containerName="payload-pre-processing"
Sep 18 13:57:34.801101 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.800971    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="359f8ffd-44ac-46ce-a416-75d25a893b27" containerName="payload-pre-processing"
Sep 18 13:57:34.801173 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.801112    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="359f8ffd-44ac-46ce-a416-75d25a893b27" containerName="payload-pre-processing"
Sep 18 13:57:34.806026 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.806007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:34.808805 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.808784    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-feff2644-serving-cert\""
Sep 18 13:57:34.813256 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.813235    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"]
Sep 18 13:57:34.942351 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.942315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cdb2be-d875-4b3e-bc8a-51af4286bd10-openshift-service-ca\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:34.942351 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.942350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-metrics-certs\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:34.942590 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.942379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj9bj\" (UniqueName: \"kubernetes.io/projected/82cdb2be-d875-4b3e-bc8a-51af4286bd10-kube-api-access-jj9bj\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:34.942590 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:34.942443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-maas-api-tls\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:35.043581 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.043551    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj9bj\" (UniqueName: \"kubernetes.io/projected/82cdb2be-d875-4b3e-bc8a-51af4286bd10-kube-api-access-jj9bj\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:35.043731 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.043589    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-maas-api-tls\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:35.043731 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.043655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cdb2be-d875-4b3e-bc8a-51af4286bd10-openshift-service-ca\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:35.043731 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.043671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-metrics-certs\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:35.044457 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.044439    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cdb2be-d875-4b3e-bc8a-51af4286bd10-openshift-service-ca\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:35.046221 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.046201    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-metrics-certs\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:35.046284 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.046228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-maas-api-tls\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:35.052392 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.052347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj9bj\" (UniqueName: \"kubernetes.io/projected/82cdb2be-d875-4b3e-bc8a-51af4286bd10-kube-api-access-jj9bj\") pod \"maas-api-e2e-ait-feff2644-5bdc455697-jlbjj\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:35.119980 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.119950    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:35.406259 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.406238    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"]
Sep 18 13:57:35.408279 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:57:35.408251    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cdb2be_d875_4b3e_bc8a_51af4286bd10.slice/crio-c3c0a312111ba087559b4e097587fa8f2a6c6d4c117e701d65c05974b52750ab WatchSource:0}: Error finding container c3c0a312111ba087559b4e097587fa8f2a6c6d4c117e701d65c05974b52750ab: Status 404 returned error can't find the container with id c3c0a312111ba087559b4e097587fa8f2a6c6d4c117e701d65c05974b52750ab
Sep 18 13:57:35.539507 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.539478    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"]
Sep 18 13:57:35.544167 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.544147    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:57:35.547506 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.547473    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-feff2644\""
Sep 18 13:57:35.556208 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.556185    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"]
Sep 18 13:57:35.650785 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.648410    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvxq\" (UniqueName: \"kubernetes.io/projected/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-kube-api-access-gcvxq\") pod \"payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7\" (UID: \"f2e8cfd6-49be-4e97-9a7d-76c65c78203e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:57:35.650785 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.648503    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7\" (UID: \"f2e8cfd6-49be-4e97-9a7d-76c65c78203e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:57:35.749482 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.749437    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvxq\" (UniqueName: \"kubernetes.io/projected/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-kube-api-access-gcvxq\") pod \"payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7\" (UID: \"f2e8cfd6-49be-4e97-9a7d-76c65c78203e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:57:35.749659 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.749515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7\" (UID: \"f2e8cfd6-49be-4e97-9a7d-76c65c78203e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:57:35.750211 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.750195    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7\" (UID: \"f2e8cfd6-49be-4e97-9a7d-76c65c78203e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:57:35.758521 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.758503    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvxq\" (UniqueName: \"kubernetes.io/projected/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-kube-api-access-gcvxq\") pod \"payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7\" (UID: \"f2e8cfd6-49be-4e97-9a7d-76c65c78203e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:57:35.855850 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.855825    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:57:35.903619 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:35.903582    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj" event={"ID":"82cdb2be-d875-4b3e-bc8a-51af4286bd10","Type":"ContainerStarted","Data":"c3c0a312111ba087559b4e097587fa8f2a6c6d4c117e701d65c05974b52750ab"}
Sep 18 13:57:36.151836 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:36.151812    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"]
Sep 18 13:57:36.153131 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:57:36.153106    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e8cfd6_49be_4e97_9a7d_76c65c78203e.slice/crio-9197c6be0a912ab7f40975133b3f6aaf421df023b1b72180b3881024dd65887e WatchSource:0}: Error finding container 9197c6be0a912ab7f40975133b3f6aaf421df023b1b72180b3881024dd65887e: Status 404 returned error can't find the container with id 9197c6be0a912ab7f40975133b3f6aaf421df023b1b72180b3881024dd65887e
Sep 18 13:57:36.909932 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:36.909886    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7" event={"ID":"f2e8cfd6-49be-4e97-9a7d-76c65c78203e","Type":"ContainerStarted","Data":"d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef"}
Sep 18 13:57:36.910386 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:36.909964    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:57:36.910386 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:36.909989    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7" event={"ID":"f2e8cfd6-49be-4e97-9a7d-76c65c78203e","Type":"ContainerStarted","Data":"9197c6be0a912ab7f40975133b3f6aaf421df023b1b72180b3881024dd65887e"}
Sep 18 13:57:36.911620 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:36.911598    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj" event={"ID":"82cdb2be-d875-4b3e-bc8a-51af4286bd10","Type":"ContainerStarted","Data":"fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07"}
Sep 18 13:57:36.911725 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:36.911644    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:36.929359 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:36.929323    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7" podStartSLOduration=1.9293102100000001 podStartE2EDuration="1.92931021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:57:36.928329086 +0000 UTC m=+798.830785273" watchObservedRunningTime="2026-09-18 13:57:36.92931021 +0000 UTC m=+798.831766443"
Sep 18 13:57:36.946438 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:36.946398    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj" podStartSLOduration=2.385807353 podStartE2EDuration="2.946382604s" totalImagesPullingTime="560.575251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:57:36.945975182 +0000 UTC m=+798.848431399" watchObservedRunningTime="2026-09-18 13:57:36.946382604 +0000 UTC m=+798.848838793"
Sep 18 13:57:42.922308 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:42.922228    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:57:47.919171 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:57:47.919140    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:58:04.460108 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.460051    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"]
Sep 18 13:58:04.460677 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.460282    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj" podUID="82cdb2be-d875-4b3e-bc8a-51af4286bd10" containerName="maas-api" containerID="cri-o://fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07" gracePeriod=30
Sep 18 13:58:04.539396 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.539360    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"]
Sep 18 13:58:04.539663 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.539630    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7" podUID="f2e8cfd6-49be-4e97-9a7d-76c65c78203e" containerName="payload-pre-processing" containerID="cri-o://d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef" gracePeriod=30
Sep 18 13:58:04.804287 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.804258    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:58:04.841292 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.841270    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:58:04.910637 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.910609    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-plugins-config-volume\") pod \"f2e8cfd6-49be-4e97-9a7d-76c65c78203e\" (UID: \"f2e8cfd6-49be-4e97-9a7d-76c65c78203e\") "
Sep 18 13:58:04.910813 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.910686    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-metrics-certs\") pod \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") "
Sep 18 13:58:04.910813 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.910763    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-maas-api-tls\") pod \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") "
Sep 18 13:58:04.910813 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.910787    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcvxq\" (UniqueName: \"kubernetes.io/projected/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-kube-api-access-gcvxq\") pod \"f2e8cfd6-49be-4e97-9a7d-76c65c78203e\" (UID: \"f2e8cfd6-49be-4e97-9a7d-76c65c78203e\") "
Sep 18 13:58:04.910978 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.910814    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cdb2be-d875-4b3e-bc8a-51af4286bd10-openshift-service-ca\") pod \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") "
Sep 18 13:58:04.910978 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.910854    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj9bj\" (UniqueName: \"kubernetes.io/projected/82cdb2be-d875-4b3e-bc8a-51af4286bd10-kube-api-access-jj9bj\") pod \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\" (UID: \"82cdb2be-d875-4b3e-bc8a-51af4286bd10\") "
Sep 18 13:58:04.911107 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.910984    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2e8cfd6-49be-4e97-9a7d-76c65c78203e" (UID: "f2e8cfd6-49be-4e97-9a7d-76c65c78203e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:58:04.911383 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.911357    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:04.911688 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.911383    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cdb2be-d875-4b3e-bc8a-51af4286bd10-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82cdb2be-d875-4b3e-bc8a-51af4286bd10" (UID: "82cdb2be-d875-4b3e-bc8a-51af4286bd10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:58:04.913502 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.913484    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "82cdb2be-d875-4b3e-bc8a-51af4286bd10" (UID: "82cdb2be-d875-4b3e-bc8a-51af4286bd10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:04.913694 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.913668    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82cdb2be-d875-4b3e-bc8a-51af4286bd10" (UID: "82cdb2be-d875-4b3e-bc8a-51af4286bd10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:04.913694 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.913682    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cdb2be-d875-4b3e-bc8a-51af4286bd10-kube-api-access-jj9bj" (OuterVolumeSpecName: "kube-api-access-jj9bj") pod "82cdb2be-d875-4b3e-bc8a-51af4286bd10" (UID: "82cdb2be-d875-4b3e-bc8a-51af4286bd10"). InnerVolumeSpecName "kube-api-access-jj9bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:04.913828 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:04.913696    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-kube-api-access-gcvxq" (OuterVolumeSpecName: "kube-api-access-gcvxq") pod "f2e8cfd6-49be-4e97-9a7d-76c65c78203e" (UID: "f2e8cfd6-49be-4e97-9a7d-76c65c78203e"). InnerVolumeSpecName "kube-api-access-gcvxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:05.012756 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.012662    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.012756 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.012693    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cdb2be-d875-4b3e-bc8a-51af4286bd10-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.012756 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.012706    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcvxq\" (UniqueName: \"kubernetes.io/projected/f2e8cfd6-49be-4e97-9a7d-76c65c78203e-kube-api-access-gcvxq\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.012756 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.012718    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cdb2be-d875-4b3e-bc8a-51af4286bd10-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.012756 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.012729    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj9bj\" (UniqueName: \"kubernetes.io/projected/82cdb2be-d875-4b3e-bc8a-51af4286bd10-kube-api-access-jj9bj\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.055160 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.055127    2569 generic.go:356] "Generic (PLEG): container finished" podID="f2e8cfd6-49be-4e97-9a7d-76c65c78203e" containerID="d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef" exitCode=0
Sep 18 13:58:05.055346 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.055176    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"
Sep 18 13:58:05.055346 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.055227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7" event={"ID":"f2e8cfd6-49be-4e97-9a7d-76c65c78203e","Type":"ContainerDied","Data":"d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef"}
Sep 18 13:58:05.055346 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.055290    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7" event={"ID":"f2e8cfd6-49be-4e97-9a7d-76c65c78203e","Type":"ContainerDied","Data":"9197c6be0a912ab7f40975133b3f6aaf421df023b1b72180b3881024dd65887e"}
Sep 18 13:58:05.055346 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.055320    2569 scope.go:117] "RemoveContainer" containerID="d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef"
Sep 18 13:58:05.056896 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.056870    2569 generic.go:356] "Generic (PLEG): container finished" podID="82cdb2be-d875-4b3e-bc8a-51af4286bd10" containerID="fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07" exitCode=0
Sep 18 13:58:05.057015 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.056914    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"
Sep 18 13:58:05.057015 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.056918    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj" event={"ID":"82cdb2be-d875-4b3e-bc8a-51af4286bd10","Type":"ContainerDied","Data":"fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07"}
Sep 18 13:58:05.057015 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.056960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj" event={"ID":"82cdb2be-d875-4b3e-bc8a-51af4286bd10","Type":"ContainerDied","Data":"c3c0a312111ba087559b4e097587fa8f2a6c6d4c117e701d65c05974b52750ab"}
Sep 18 13:58:05.065980 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.065959    2569 scope.go:117] "RemoveContainer" containerID="d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef"
Sep 18 13:58:05.066276 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:58:05.066235    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef\": container with ID starting with d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef not found: ID does not exist" containerID="d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef"
Sep 18 13:58:05.066276 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.066263    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef"} err="failed to get container status \"d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef\": rpc error: code = NotFound desc = could not find container \"d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef\": container with ID starting with d4e8caba04d1f857e9ce2c281fa6f6e093f007df7f5122955fde62cbc162f8ef not found: ID does not exist"
Sep 18 13:58:05.066393 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.066278    2569 scope.go:117] "RemoveContainer" containerID="fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07"
Sep 18 13:58:05.074501 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.074486    2569 scope.go:117] "RemoveContainer" containerID="fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07"
Sep 18 13:58:05.074777 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:58:05.074758    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07\": container with ID starting with fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07 not found: ID does not exist" containerID="fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07"
Sep 18 13:58:05.074883 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.074782    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07"} err="failed to get container status \"fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07\": rpc error: code = NotFound desc = could not find container \"fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07\": container with ID starting with fedb5f04e453523fa9737ed193462b609ea845f72df742d993d13bb29663fe07 not found: ID does not exist"
Sep 18 13:58:05.081253 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.081234    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"]
Sep 18 13:58:05.085720 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.085702    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-feff2644-7c6f4fbd7d-z8wn7"]
Sep 18 13:58:05.097309 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.097292    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"]
Sep 18 13:58:05.103348 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:05.103328    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-feff2644-5bdc455697-jlbjj"]
Sep 18 13:58:06.488359 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.488320    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82cdb2be-d875-4b3e-bc8a-51af4286bd10" path="/var/lib/kubelet/pods/82cdb2be-d875-4b3e-bc8a-51af4286bd10/volumes"
Sep 18 13:58:06.488954 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.488931    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e8cfd6-49be-4e97-9a7d-76c65c78203e" path="/var/lib/kubelet/pods/f2e8cfd6-49be-4e97-9a7d-76c65c78203e/volumes"
Sep 18 13:58:06.568515 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.568485    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"]
Sep 18 13:58:06.568931 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.568914    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82cdb2be-d875-4b3e-bc8a-51af4286bd10" containerName="maas-api"
Sep 18 13:58:06.569011 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.568934    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="82cdb2be-d875-4b3e-bc8a-51af4286bd10" containerName="maas-api"
Sep 18 13:58:06.569011 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.568969    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e8cfd6-49be-4e97-9a7d-76c65c78203e" containerName="payload-pre-processing"
Sep 18 13:58:06.569011 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.568978    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e8cfd6-49be-4e97-9a7d-76c65c78203e" containerName="payload-pre-processing"
Sep 18 13:58:06.569194 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.569102    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="82cdb2be-d875-4b3e-bc8a-51af4286bd10" containerName="maas-api"
Sep 18 13:58:06.569194 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.569114    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e8cfd6-49be-4e97-9a7d-76c65c78203e" containerName="payload-pre-processing"
Sep 18 13:58:06.573841 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.573822    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.577830 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.577806    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-df538fee-openshift-default-dockercfg-dcqk6\""
Sep 18 13:58:06.586299 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.586262    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"]
Sep 18 13:58:06.628805 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.628773    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istiod-ca-cert\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.628805 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.628809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8c5w\" (UniqueName: \"kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-kube-api-access-t8c5w\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.629013 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.628840    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-data\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.629013 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.628882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-token\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.629013 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.628914    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-envoy\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.629013 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.628941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-credential-socket\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.629013 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.628994    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-socket\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.629013 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.629008    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-certs\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.629351 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.629032    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-podinfo\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.730442 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.730405    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-credential-socket\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.730627 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.730466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-socket\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.730627 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.730489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-certs\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.730627 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.730511    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-podinfo\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.730627 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.730594    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istiod-ca-cert\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.730627 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.730623    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8c5w\" (UniqueName: \"kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-kube-api-access-t8c5w\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.730934 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.730663    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-data\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.730934 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.730691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-token\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.730934 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.730731    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-envoy\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.730934 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.730910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-credential-socket\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.731195 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.731167    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-socket\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.731260 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.731206    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-data\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.731313 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.731260    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-certs\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.731468 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.731450    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istiod-ca-cert\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.733356 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.733331    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-envoy\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.733804 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.733785    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-podinfo\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.757617 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.757565    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-token\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.757725 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.757646    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8c5w\" (UniqueName: \"kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-kube-api-access-t8c5w\") pod \"e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") " pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:06.885016 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:06.884980    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:07.164434 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:07.164411    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"]
Sep 18 13:58:07.166593 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:58:07.166566    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdddc01_6db3_4bd9_8225_eea063b2f5b4.slice/crio-e237ffbee42f813e1bad0be2489404730825a78b68a154d66a0720caeca97cb2 WatchSource:0}: Error finding container e237ffbee42f813e1bad0be2489404730825a78b68a154d66a0720caeca97cb2: Status 404 returned error can't find the container with id e237ffbee42f813e1bad0be2489404730825a78b68a154d66a0720caeca97cb2
Sep 18 13:58:07.168873 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:07.168841    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:58:07.168958 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:07.168902    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:58:07.168958 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:07.168929    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:58:08.076053 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:08.076016    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" event={"ID":"7fdddc01-6db3-4bd9-8225-eea063b2f5b4","Type":"ContainerStarted","Data":"5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0"}
Sep 18 13:58:08.076466 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:08.076079    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" event={"ID":"7fdddc01-6db3-4bd9-8225-eea063b2f5b4","Type":"ContainerStarted","Data":"e237ffbee42f813e1bad0be2489404730825a78b68a154d66a0720caeca97cb2"}
Sep 18 13:58:08.102628 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:08.102589    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" podStartSLOduration=2.102572406 podStartE2EDuration="2.102572406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:08.100682577 +0000 UTC m=+830.003138763" watchObservedRunningTime="2026-09-18 13:58:08.102572406 +0000 UTC m=+830.005028592"
Sep 18 13:58:08.885387 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:08.885348    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:08.890017 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:08.889994    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:09.081946 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:09.081918    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:09.082759 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:09.082743    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:13.491963 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.491931    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"]
Sep 18 13:58:13.496442 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.496424    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.499368 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.499339    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-df538fee-serving-cert\""
Sep 18 13:58:13.508190 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.508163    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"]
Sep 18 13:58:13.592359 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.592330    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd98e22-a65c-4998-b339-5fc6bc069542-openshift-service-ca\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.592481 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.592366    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-metrics-certs\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.592481 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.592385    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-maas-api-tls\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.592566 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.592540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzgwn\" (UniqueName: \"kubernetes.io/projected/ebd98e22-a65c-4998-b339-5fc6bc069542-kube-api-access-bzgwn\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.693911 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.693879    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzgwn\" (UniqueName: \"kubernetes.io/projected/ebd98e22-a65c-4998-b339-5fc6bc069542-kube-api-access-bzgwn\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.694098 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.693946    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd98e22-a65c-4998-b339-5fc6bc069542-openshift-service-ca\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.694098 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.693969    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-metrics-certs\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.694098 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.693985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-maas-api-tls\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.694635 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.694617    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd98e22-a65c-4998-b339-5fc6bc069542-openshift-service-ca\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.696595 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.696573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-metrics-certs\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.696736 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.696637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-maas-api-tls\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.705978 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.705953    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzgwn\" (UniqueName: \"kubernetes.io/projected/ebd98e22-a65c-4998-b339-5fc6bc069542-kube-api-access-bzgwn\") pod \"maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:13.811626 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:13.811569    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:14.085502 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:14.085479    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"]
Sep 18 13:58:14.088085 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:58:14.088043    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd98e22_a65c_4998_b339_5fc6bc069542.slice/crio-a97e3c59102d774c80614e33e1343a42c9d7c09c55252f7465b3c29578274d8c WatchSource:0}: Error finding container a97e3c59102d774c80614e33e1343a42c9d7c09c55252f7465b3c29578274d8c: Status 404 returned error can't find the container with id a97e3c59102d774c80614e33e1343a42c9d7c09c55252f7465b3c29578274d8c
Sep 18 13:58:14.106576 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:14.106549    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm" event={"ID":"ebd98e22-a65c-4998-b339-5fc6bc069542","Type":"ContainerStarted","Data":"a97e3c59102d774c80614e33e1343a42c9d7c09c55252f7465b3c29578274d8c"}
Sep 18 13:58:15.116792 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:15.116752    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm" event={"ID":"ebd98e22-a65c-4998-b339-5fc6bc069542","Type":"ContainerStarted","Data":"c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0"}
Sep 18 13:58:15.117215 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:15.116960    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:15.136298 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:15.136248    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm" podStartSLOduration=1.7816001209999999 podStartE2EDuration="2.136223878s" totalImagesPullingTime="354.623757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:15.133952999 +0000 UTC m=+837.036409185" watchObservedRunningTime="2026-09-18 13:58:15.136223878 +0000 UTC m=+837.038680098"
Sep 18 13:58:21.127062 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:21.127031    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:35.217100 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:35.217051    2569 generic.go:356] "Generic (PLEG): container finished" podID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerID="5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0" exitCode=0
Sep 18 13:58:35.217505 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:35.217125    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" event={"ID":"7fdddc01-6db3-4bd9-8225-eea063b2f5b4","Type":"ContainerDied","Data":"5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0"}
Sep 18 13:58:35.217505 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:35.217460    2569 scope.go:117] "RemoveContainer" containerID="5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0"
Sep 18 13:58:35.218499 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:35.218469    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:58:35.218609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:35.218539    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:58:35.218609 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:35.218571    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:58:36.093590 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.093553    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"]
Sep 18 13:58:36.093878 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.093799    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt" podUID="859f21d1-3103-44ee-998a-7c0c8fbcac32" containerName="maas-api" containerID="cri-o://75954e6e330ec8bbcc0c5664a23e7acbc87192bce9ab0f6ac17ee352e9e88b55" gracePeriod=30
Sep 18 13:58:36.226501 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.226468    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" event={"ID":"7fdddc01-6db3-4bd9-8225-eea063b2f5b4","Type":"ContainerStarted","Data":"53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92"}
Sep 18 13:58:36.228364 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.228340    2569 generic.go:356] "Generic (PLEG): container finished" podID="859f21d1-3103-44ee-998a-7c0c8fbcac32" containerID="75954e6e330ec8bbcc0c5664a23e7acbc87192bce9ab0f6ac17ee352e9e88b55" exitCode=0
Sep 18 13:58:36.228483 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.228413    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt" event={"ID":"859f21d1-3103-44ee-998a-7c0c8fbcac32","Type":"ContainerDied","Data":"75954e6e330ec8bbcc0c5664a23e7acbc87192bce9ab0f6ac17ee352e9e88b55"}
Sep 18 13:58:36.394889 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.394865    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:58:36.519604 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.519574    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wxnw\" (UniqueName: \"kubernetes.io/projected/859f21d1-3103-44ee-998a-7c0c8fbcac32-kube-api-access-6wxnw\") pod \"859f21d1-3103-44ee-998a-7c0c8fbcac32\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") "
Sep 18 13:58:36.519750 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.519614    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-maas-api-tls\") pod \"859f21d1-3103-44ee-998a-7c0c8fbcac32\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") "
Sep 18 13:58:36.519750 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.519693    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-metrics-certs\") pod \"859f21d1-3103-44ee-998a-7c0c8fbcac32\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") "
Sep 18 13:58:36.519750 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.519740    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/859f21d1-3103-44ee-998a-7c0c8fbcac32-openshift-service-ca\") pod \"859f21d1-3103-44ee-998a-7c0c8fbcac32\" (UID: \"859f21d1-3103-44ee-998a-7c0c8fbcac32\") "
Sep 18 13:58:36.520190 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.520161    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859f21d1-3103-44ee-998a-7c0c8fbcac32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "859f21d1-3103-44ee-998a-7c0c8fbcac32" (UID: "859f21d1-3103-44ee-998a-7c0c8fbcac32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:58:36.521902 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.521875    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "859f21d1-3103-44ee-998a-7c0c8fbcac32" (UID: "859f21d1-3103-44ee-998a-7c0c8fbcac32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:36.522018 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.521955    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859f21d1-3103-44ee-998a-7c0c8fbcac32-kube-api-access-6wxnw" (OuterVolumeSpecName: "kube-api-access-6wxnw") pod "859f21d1-3103-44ee-998a-7c0c8fbcac32" (UID: "859f21d1-3103-44ee-998a-7c0c8fbcac32"). InnerVolumeSpecName "kube-api-access-6wxnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:36.522018 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.521977    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "859f21d1-3103-44ee-998a-7c0c8fbcac32" (UID: "859f21d1-3103-44ee-998a-7c0c8fbcac32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:36.620493 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.620459    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:36.620493 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.620490    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/859f21d1-3103-44ee-998a-7c0c8fbcac32-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:36.620493 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.620500    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wxnw\" (UniqueName: \"kubernetes.io/projected/859f21d1-3103-44ee-998a-7c0c8fbcac32-kube-api-access-6wxnw\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:36.620790 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.620510    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/859f21d1-3103-44ee-998a-7c0c8fbcac32-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:36.885434 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.885397    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:36.886668 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.886635    2569 patch_prober.go:28] interesting pod/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 13:58:36.886813 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:36.886686    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 13:58:37.234600 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:37.234562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt" event={"ID":"859f21d1-3103-44ee-998a-7c0c8fbcac32","Type":"ContainerDied","Data":"3bda572d6afe70f30670faaf03f196ecf0b520046654af5021ce5f8299a45d96"}
Sep 18 13:58:37.235087 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:37.234572    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"
Sep 18 13:58:37.235087 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:37.234638    2569 scope.go:117] "RemoveContainer" containerID="75954e6e330ec8bbcc0c5664a23e7acbc87192bce9ab0f6ac17ee352e9e88b55"
Sep 18 13:58:37.283828 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:37.283803    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"]
Sep 18 13:58:37.290405 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:37.290380    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-70f06e-6cd57685f5-sklxt"]
Sep 18 13:58:37.886246 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:37.886199    2569 patch_prober.go:28] interesting pod/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 13:58:37.886440 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:37.886279    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 13:58:38.486243 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:38.486212    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="859f21d1-3103-44ee-998a-7c0c8fbcac32" path="/var/lib/kubelet/pods/859f21d1-3103-44ee-998a-7c0c8fbcac32/volumes"
Sep 18 13:58:38.886316 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:38.886236    2569 patch_prober.go:28] interesting pod/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 13:58:38.886316 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:38.886285    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 13:58:39.082263 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:39.082229    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:39.885980 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:39.885951    2569 patch_prober.go:28] interesting pod/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 13:58:39.886390 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:39.886007    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 13:58:40.885542 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:40.885497    2569 patch_prober.go:28] interesting pod/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 18 13:58:40.885802 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:40.885561    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 18 13:58:41.676343 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.676311    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"]
Sep 18 13:58:41.676808 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.676547    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm" podUID="ebd98e22-a65c-4998-b339-5fc6bc069542" containerName="maas-api" containerID="cri-o://c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0" gracePeriod=30
Sep 18 13:58:41.890916 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.890888    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:41.892098 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.892061    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:41.931844 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.931780    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:41.975617 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.975580    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd98e22-a65c-4998-b339-5fc6bc069542-openshift-service-ca\") pod \"ebd98e22-a65c-4998-b339-5fc6bc069542\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") "
Sep 18 13:58:41.975816 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.975755    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzgwn\" (UniqueName: \"kubernetes.io/projected/ebd98e22-a65c-4998-b339-5fc6bc069542-kube-api-access-bzgwn\") pod \"ebd98e22-a65c-4998-b339-5fc6bc069542\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") "
Sep 18 13:58:41.975816 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.975794    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-metrics-certs\") pod \"ebd98e22-a65c-4998-b339-5fc6bc069542\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") "
Sep 18 13:58:41.975941 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.975823    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-maas-api-tls\") pod \"ebd98e22-a65c-4998-b339-5fc6bc069542\" (UID: \"ebd98e22-a65c-4998-b339-5fc6bc069542\") "
Sep 18 13:58:41.976827 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.976787    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd98e22-a65c-4998-b339-5fc6bc069542-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebd98e22-a65c-4998-b339-5fc6bc069542" (UID: "ebd98e22-a65c-4998-b339-5fc6bc069542"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:58:41.980496 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.980465    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd98e22-a65c-4998-b339-5fc6bc069542-kube-api-access-bzgwn" (OuterVolumeSpecName: "kube-api-access-bzgwn") pod "ebd98e22-a65c-4998-b339-5fc6bc069542" (UID: "ebd98e22-a65c-4998-b339-5fc6bc069542"). InnerVolumeSpecName "kube-api-access-bzgwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:41.980988 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.980902    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ebd98e22-a65c-4998-b339-5fc6bc069542" (UID: "ebd98e22-a65c-4998-b339-5fc6bc069542"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:41.982581 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:41.982552    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebd98e22-a65c-4998-b339-5fc6bc069542" (UID: "ebd98e22-a65c-4998-b339-5fc6bc069542"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:42.077485 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.077445    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzgwn\" (UniqueName: \"kubernetes.io/projected/ebd98e22-a65c-4998-b339-5fc6bc069542-kube-api-access-bzgwn\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:42.077485 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.077483    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:42.077698 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.077497    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd98e22-a65c-4998-b339-5fc6bc069542-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:42.077698 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.077508    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd98e22-a65c-4998-b339-5fc6bc069542-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:42.107730 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.106611    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"]
Sep 18 13:58:42.269956 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.269922    2569 generic.go:356] "Generic (PLEG): container finished" podID="ebd98e22-a65c-4998-b339-5fc6bc069542" containerID="c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0" exitCode=0
Sep 18 13:58:42.270176 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.269975    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"
Sep 18 13:58:42.270176 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.269999    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm" event={"ID":"ebd98e22-a65c-4998-b339-5fc6bc069542","Type":"ContainerDied","Data":"c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0"}
Sep 18 13:58:42.270176 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.270052    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm" event={"ID":"ebd98e22-a65c-4998-b339-5fc6bc069542","Type":"ContainerDied","Data":"a97e3c59102d774c80614e33e1343a42c9d7c09c55252f7465b3c29578274d8c"}
Sep 18 13:58:42.270176 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.270097    2569 scope.go:117] "RemoveContainer" containerID="c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0"
Sep 18 13:58:42.270457 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.270439    2569 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" secret="" err="secret \"e2e-derive-df538fee-openshift-default-dockercfg-dcqk6\" not found"
Sep 18 13:58:42.279817 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.279798    2569 scope.go:117] "RemoveContainer" containerID="c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0"
Sep 18 13:58:42.280154 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:58:42.280134    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0\": container with ID starting with c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0 not found: ID does not exist" containerID="c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0"
Sep 18 13:58:42.280241 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.280163    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0"} err="failed to get container status \"c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0\": rpc error: code = NotFound desc = could not find container \"c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0\": container with ID starting with c959211ea8e3b328792719000baef42844b00bb1926b323c6460d2cae760b7f0 not found: ID does not exist"
Sep 18 13:58:42.294507 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.294476    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"]
Sep 18 13:58:42.298758 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.298728    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df538fee-5b6b8989cf-lb7jm"]
Sep 18 13:58:42.486647 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:42.486610    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd98e22-a65c-4998-b339-5fc6bc069542" path="/var/lib/kubelet/pods/ebd98e22-a65c-4998-b339-5fc6bc069542/volumes"
Sep 18 13:58:43.277572 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:43.277540    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy" containerID="cri-o://53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92" gracePeriod=30
Sep 18 13:58:48.534392 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.534369    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:48.634539 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634479    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-envoy\") pod \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") "
Sep 18 13:58:48.634539 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634519    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8c5w\" (UniqueName: \"kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-kube-api-access-t8c5w\") pod \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") "
Sep 18 13:58:48.634698 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634554    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-credential-socket\") pod \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") "
Sep 18 13:58:48.634698 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634569    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-socket\") pod \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") "
Sep 18 13:58:48.634698 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634588    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-podinfo\") pod \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") "
Sep 18 13:58:48.634698 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634605    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-data\") pod \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") "
Sep 18 13:58:48.634698 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634647    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-token\") pod \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") "
Sep 18 13:58:48.634698 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634678    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istiod-ca-cert\") pod \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") "
Sep 18 13:58:48.634973 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634740    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-certs\") pod \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\" (UID: \"7fdddc01-6db3-4bd9-8225-eea063b2f5b4\") "
Sep 18 13:58:48.634973 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634874    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7fdddc01-6db3-4bd9-8225-eea063b2f5b4" (UID: "7fdddc01-6db3-4bd9-8225-eea063b2f5b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:48.635106 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634981    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7fdddc01-6db3-4bd9-8225-eea063b2f5b4" (UID: "7fdddc01-6db3-4bd9-8225-eea063b2f5b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:48.635106 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.634994    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-data" (OuterVolumeSpecName: "istio-data") pod "7fdddc01-6db3-4bd9-8225-eea063b2f5b4" (UID: "7fdddc01-6db3-4bd9-8225-eea063b2f5b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:48.635204 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.635170    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7fdddc01-6db3-4bd9-8225-eea063b2f5b4" (UID: "7fdddc01-6db3-4bd9-8225-eea063b2f5b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:48.635266 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.635249    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-credential-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:48.635339 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.635274    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:48.635339 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.635290    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-data\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:48.635523 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.635503    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7fdddc01-6db3-4bd9-8225-eea063b2f5b4" (UID: "7fdddc01-6db3-4bd9-8225-eea063b2f5b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:58:48.636967 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.636938    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-kube-api-access-t8c5w" (OuterVolumeSpecName: "kube-api-access-t8c5w") pod "7fdddc01-6db3-4bd9-8225-eea063b2f5b4" (UID: "7fdddc01-6db3-4bd9-8225-eea063b2f5b4"). InnerVolumeSpecName "kube-api-access-t8c5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:48.636967 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.636962    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7fdddc01-6db3-4bd9-8225-eea063b2f5b4" (UID: "7fdddc01-6db3-4bd9-8225-eea063b2f5b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:48.637179 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.637143    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7fdddc01-6db3-4bd9-8225-eea063b2f5b4" (UID: "7fdddc01-6db3-4bd9-8225-eea063b2f5b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:58:48.637426 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.637400    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-token" (OuterVolumeSpecName: "istio-token") pod "7fdddc01-6db3-4bd9-8225-eea063b2f5b4" (UID: "7fdddc01-6db3-4bd9-8225-eea063b2f5b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:48.736579 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.736553    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-podinfo\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:48.736579 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.736575    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-token\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:48.736579 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.736584    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istiod-ca-cert\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:48.736750 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.736591    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-workload-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:48.736750 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.736599    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-istio-envoy\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:48.736750 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:48.736606    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8c5w\" (UniqueName: \"kubernetes.io/projected/7fdddc01-6db3-4bd9-8225-eea063b2f5b4-kube-api-access-t8c5w\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 13:58:49.315712 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.315674    2569 generic.go:356] "Generic (PLEG): container finished" podID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerID="53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92" exitCode=0
Sep 18 13:58:49.315898 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.315731    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"
Sep 18 13:58:49.315898 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.315771    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" event={"ID":"7fdddc01-6db3-4bd9-8225-eea063b2f5b4","Type":"ContainerDied","Data":"53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92"}
Sep 18 13:58:49.315898 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.315844    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p" event={"ID":"7fdddc01-6db3-4bd9-8225-eea063b2f5b4","Type":"ContainerDied","Data":"e237ffbee42f813e1bad0be2489404730825a78b68a154d66a0720caeca97cb2"}
Sep 18 13:58:49.315898 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.315872    2569 scope.go:117] "RemoveContainer" containerID="53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92"
Sep 18 13:58:49.326835 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.326816    2569 scope.go:117] "RemoveContainer" containerID="5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0"
Sep 18 13:58:49.336216 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.336198    2569 scope.go:117] "RemoveContainer" containerID="53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92"
Sep 18 13:58:49.336496 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:58:49.336476    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92\": container with ID starting with 53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92 not found: ID does not exist" containerID="53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92"
Sep 18 13:58:49.336546 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.336507    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92"} err="failed to get container status \"53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92\": rpc error: code = NotFound desc = could not find container \"53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92\": container with ID starting with 53e915634d1d080babc392c90282f829eb11cca3112769e15913d057cc5a9e92 not found: ID does not exist"
Sep 18 13:58:49.336546 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.336534    2569 scope.go:117] "RemoveContainer" containerID="5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0"
Sep 18 13:58:49.336784 ip-10-0-143-135 kubenswrapper[2569]: E0918 13:58:49.336766    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0\": container with ID starting with 5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0 not found: ID does not exist" containerID="5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0"
Sep 18 13:58:49.336843 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.336794    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0"} err="failed to get container status \"5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0\": rpc error: code = NotFound desc = could not find container \"5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0\": container with ID starting with 5c2a77a80e83cc15d5f83d20c2ed714a945c090b28f4672def6d544ed559a1f0 not found: ID does not exist"
Sep 18 13:58:49.361436 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.361407    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"]
Sep 18 13:58:49.364685 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:49.364661    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-df538fee-openshift-default-77b5b8d9d4-vw24p"]
Sep 18 13:58:50.485895 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:58:50.485867    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" path="/var/lib/kubelet/pods/7fdddc01-6db3-4bd9-8225-eea063b2f5b4/volumes"
Sep 18 13:59:18.495153 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:18.495127    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:59:18.499545 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:18.499526    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 13:59:27.557498 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.557453    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"]
Sep 18 13:59:27.558093 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558044    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="859f21d1-3103-44ee-998a-7c0c8fbcac32" containerName="maas-api"
Sep 18 13:59:27.558093 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558065    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="859f21d1-3103-44ee-998a-7c0c8fbcac32" containerName="maas-api"
Sep 18 13:59:27.558266 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558098    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd98e22-a65c-4998-b339-5fc6bc069542" containerName="maas-api"
Sep 18 13:59:27.558266 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558108    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd98e22-a65c-4998-b339-5fc6bc069542" containerName="maas-api"
Sep 18 13:59:27.558266 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558115    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy"
Sep 18 13:59:27.558266 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558120    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy"
Sep 18 13:59:27.558266 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558130    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy"
Sep 18 13:59:27.558266 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558135    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy"
Sep 18 13:59:27.558266 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558204    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy"
Sep 18 13:59:27.558266 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558212    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="859f21d1-3103-44ee-998a-7c0c8fbcac32" containerName="maas-api"
Sep 18 13:59:27.558266 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.558220    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd98e22-a65c-4998-b339-5fc6bc069542" containerName="maas-api"
Sep 18 13:59:27.562605 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.562588    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 13:59:27.565448 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.565431    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0aac26\""
Sep 18 13:59:27.572024 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.571996    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"]
Sep 18 13:59:27.603561 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.603538    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"]
Sep 18 13:59:27.603934 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.603922    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fdddc01-6db3-4bd9-8225-eea063b2f5b4" containerName="istio-proxy"
Sep 18 13:59:27.606751 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.606737    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 13:59:27.609414 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.609395    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0aac26-dockercfg-q6mns\""
Sep 18 13:59:27.617806 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.617784    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"]
Sep 18 13:59:27.753681 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.753647    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk\" (UID: \"f7b0e5ab-032a-466b-b537-e5c13dff5f1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 13:59:27.753843 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.753690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7j2g\" (UniqueName: \"kubernetes.io/projected/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-kube-api-access-k7j2g\") pod \"payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk\" (UID: \"f7b0e5ab-032a-466b-b537-e5c13dff5f1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 13:59:27.753843 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.753737    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89\" (UID: \"b16e2b5a-cf53-430c-81a4-0de5919ab4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 13:59:27.753843 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.753830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdd27\" (UniqueName: \"kubernetes.io/projected/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-kube-api-access-bdd27\") pod \"payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89\" (UID: \"b16e2b5a-cf53-430c-81a4-0de5919ab4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 13:59:27.854443 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.854371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk\" (UID: \"f7b0e5ab-032a-466b-b537-e5c13dff5f1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 13:59:27.854443 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.854420    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7j2g\" (UniqueName: \"kubernetes.io/projected/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-kube-api-access-k7j2g\") pod \"payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk\" (UID: \"f7b0e5ab-032a-466b-b537-e5c13dff5f1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 13:59:27.854633 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.854478    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89\" (UID: \"b16e2b5a-cf53-430c-81a4-0de5919ab4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 13:59:27.854633 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.854545    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdd27\" (UniqueName: \"kubernetes.io/projected/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-kube-api-access-bdd27\") pod \"payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89\" (UID: \"b16e2b5a-cf53-430c-81a4-0de5919ab4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 13:59:27.855150 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.855131    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk\" (UID: \"f7b0e5ab-032a-466b-b537-e5c13dff5f1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 13:59:27.855196 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.855175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89\" (UID: \"b16e2b5a-cf53-430c-81a4-0de5919ab4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 13:59:27.864534 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.864510    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7j2g\" (UniqueName: \"kubernetes.io/projected/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-kube-api-access-k7j2g\") pod \"payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk\" (UID: \"f7b0e5ab-032a-466b-b537-e5c13dff5f1e\") " pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 13:59:27.864671 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.864627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdd27\" (UniqueName: \"kubernetes.io/projected/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-kube-api-access-bdd27\") pod \"payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89\" (UID: \"b16e2b5a-cf53-430c-81a4-0de5919ab4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 13:59:27.873378 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.873356    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 13:59:27.918172 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:27.918148    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 13:59:28.149942 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:28.149921    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"]
Sep 18 13:59:28.151311 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:59:28.151289    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16e2b5a_cf53_430c_81a4_0de5919ab4fb.slice/crio-f0dd5e09051f69b7193619e4882b352d3d68213fd5ff070a019d9353a1bbd4f1 WatchSource:0}: Error finding container f0dd5e09051f69b7193619e4882b352d3d68213fd5ff070a019d9353a1bbd4f1: Status 404 returned error can't find the container with id f0dd5e09051f69b7193619e4882b352d3d68213fd5ff070a019d9353a1bbd4f1
Sep 18 13:59:28.201887 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:28.201866    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"]
Sep 18 13:59:28.211154 ip-10-0-143-135 kubenswrapper[2569]: W0918 13:59:28.211127    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b0e5ab_032a_466b_b537_e5c13dff5f1e.slice/crio-0fada1f34aa5fc8017ccea3e01756c9e6040146408f22af5efd46ee9fa7deb2f WatchSource:0}: Error finding container 0fada1f34aa5fc8017ccea3e01756c9e6040146408f22af5efd46ee9fa7deb2f: Status 404 returned error can't find the container with id 0fada1f34aa5fc8017ccea3e01756c9e6040146408f22af5efd46ee9fa7deb2f
Sep 18 13:59:28.520452 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:28.520410    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk" event={"ID":"f7b0e5ab-032a-466b-b537-e5c13dff5f1e","Type":"ContainerStarted","Data":"4b0153b15159bb4a35610a9605134dcb3e92e1164075c408a0b28e936873c94f"}
Sep 18 13:59:28.520649 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:28.520485    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 13:59:28.520649 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:28.520507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk" event={"ID":"f7b0e5ab-032a-466b-b537-e5c13dff5f1e","Type":"ContainerStarted","Data":"0fada1f34aa5fc8017ccea3e01756c9e6040146408f22af5efd46ee9fa7deb2f"}
Sep 18 13:59:28.522189 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:28.522160    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89" event={"ID":"b16e2b5a-cf53-430c-81a4-0de5919ab4fb","Type":"ContainerStarted","Data":"faee60d71394553a9007b51cf4f2d2b96241771ff317dd345b69135c9d072b82"}
Sep 18 13:59:28.522311 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:28.522208    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89" event={"ID":"b16e2b5a-cf53-430c-81a4-0de5919ab4fb","Type":"ContainerStarted","Data":"f0dd5e09051f69b7193619e4882b352d3d68213fd5ff070a019d9353a1bbd4f1"}
Sep 18 13:59:28.522311 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:28.522270    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 13:59:28.546443 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:28.546395    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk" podStartSLOduration=1.5463835719999999 podStartE2EDuration="1.546383572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:28.541529587 +0000 UTC m=+910.443985773" watchObservedRunningTime="2026-09-18 13:59:28.546383572 +0000 UTC m=+910.448839781"
Sep 18 13:59:28.568047 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:28.568004    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89" podStartSLOduration=1.567987237 podStartE2EDuration="1.567987237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:28.567856122 +0000 UTC m=+910.470312309" watchObservedRunningTime="2026-09-18 13:59:28.567987237 +0000 UTC m=+910.470443428"
Sep 18 13:59:39.528803 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:39.528776    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 13:59:39.529193 ip-10-0-143-135 kubenswrapper[2569]: I0918 13:59:39.528840    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 14:00:21.382939 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.382906    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"]
Sep 18 14:00:21.386716 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.386696    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.389648 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.389627    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f1bc6110-openshift-default-dockercfg-zsd7j\""
Sep 18 14:00:21.398512 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.398492    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"]
Sep 18 14:00:21.499546 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.499517    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9v46\" (UniqueName: \"kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-kube-api-access-x9v46\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.499700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.499557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-certs\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.499700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.499648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-envoy\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.499700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.499683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-istio-token\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.499878 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.499769    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94793ee8-4de8-486a-936d-3943505317d7-istio-podinfo\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.499878 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.499809    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-socket\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.499878 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.499842    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-credential-socket\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.499878 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.499872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-data\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.500029 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.499969    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94793ee8-4de8-486a-936d-3943505317d7-istiod-ca-cert\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.600917 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.600888    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-envoy\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601063 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.600929    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-istio-token\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601144 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601102    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94793ee8-4de8-486a-936d-3943505317d7-istio-podinfo\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601192 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601149    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-socket\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601320 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-credential-socket\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601410 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601340    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-data\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601410 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601386    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94793ee8-4de8-486a-936d-3943505317d7-istiod-ca-cert\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601513 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601430    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9v46\" (UniqueName: \"kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-kube-api-access-x9v46\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601513 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-certs\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601513 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601468    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-socket\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601664 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601628    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-credential-socket\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601863 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601842    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-certs\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.601991 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.601970    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-data\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.602629 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.602609    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94793ee8-4de8-486a-936d-3943505317d7-istiod-ca-cert\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.603422 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.603399    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-envoy\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.603884 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.603867    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94793ee8-4de8-486a-936d-3943505317d7-istio-podinfo\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.609611 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.609584    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-istio-token\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.610010 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.609990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9v46\" (UniqueName: \"kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-kube-api-access-x9v46\") pod \"e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") " pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:21.698798 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:21.698777    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:22.163798 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:22.163776    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"]
Sep 18 14:00:22.166329 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:00:22.166299    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94793ee8_4de8_486a_936d_3943505317d7.slice/crio-bee3de5202bf18599a516e5f4de25d1eb332c4c4805aeaed2b0eea9b6755e65e WatchSource:0}: Error finding container bee3de5202bf18599a516e5f4de25d1eb332c4c4805aeaed2b0eea9b6755e65e: Status 404 returned error can't find the container with id bee3de5202bf18599a516e5f4de25d1eb332c4c4805aeaed2b0eea9b6755e65e
Sep 18 14:00:22.168517 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:22.168485    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:00:22.168621 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:22.168548    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:00:22.168621 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:22.168575    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:00:22.790468 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:22.790430    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" event={"ID":"94793ee8-4de8-486a-936d-3943505317d7","Type":"ContainerStarted","Data":"9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c"}
Sep 18 14:00:22.790830 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:22.790497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" event={"ID":"94793ee8-4de8-486a-936d-3943505317d7","Type":"ContainerStarted","Data":"bee3de5202bf18599a516e5f4de25d1eb332c4c4805aeaed2b0eea9b6755e65e"}
Sep 18 14:00:22.838266 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:22.838225    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" podStartSLOduration=1.838210731 podStartE2EDuration="1.838210731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:22.836200015 +0000 UTC m=+964.738656201" watchObservedRunningTime="2026-09-18 14:00:22.838210731 +0000 UTC m=+964.740666917"
Sep 18 14:00:23.698931 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:23.698898    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:23.704466 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:23.704442    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:23.795758 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:23.795737    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:23.796635 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:23.796619    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:00:32.264668 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.264636    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"]
Sep 18 14:00:32.269283 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.269262    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.272091 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.272046    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f1bc6110-serving-cert\""
Sep 18 14:00:32.283198 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.283177    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"]
Sep 18 14:00:32.295228 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.295202    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-openshift-service-ca\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.295337 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.295260    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-maas-api-tls\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.295390 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.295376    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-metrics-certs\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.295427 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.295395    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhznb\" (UniqueName: \"kubernetes.io/projected/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-kube-api-access-jhznb\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.396557 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.396503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-openshift-service-ca\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.396715 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.396588    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-maas-api-tls\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.396715 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.396663    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-metrics-certs\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.396715 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.396688    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhznb\" (UniqueName: \"kubernetes.io/projected/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-kube-api-access-jhznb\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.397205 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.397186    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-openshift-service-ca\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.399103 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.399062    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-metrics-certs\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.399177 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.399145    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-maas-api-tls\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.405232 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.405214    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhznb\" (UniqueName: \"kubernetes.io/projected/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-kube-api-access-jhznb\") pod \"maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.581006 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.580931    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:32.861632 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:32.861605    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"]
Sep 18 14:00:32.864178 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:00:32.864154    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3a7012_dcf5_4128_ac00_4ca4ef1b9183.slice/crio-50a43c53af1376815830809e908962dde45390bcf59d4016c44a80ed172f0123 WatchSource:0}: Error finding container 50a43c53af1376815830809e908962dde45390bcf59d4016c44a80ed172f0123: Status 404 returned error can't find the container with id 50a43c53af1376815830809e908962dde45390bcf59d4016c44a80ed172f0123
Sep 18 14:00:33.844968 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:33.844933    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br" event={"ID":"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183","Type":"ContainerStarted","Data":"a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c"}
Sep 18 14:00:33.845357 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:33.844999    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:33.845357 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:33.845025    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br" event={"ID":"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183","Type":"ContainerStarted","Data":"50a43c53af1376815830809e908962dde45390bcf59d4016c44a80ed172f0123"}
Sep 18 14:00:33.868931 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:33.868889    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br" podStartSLOduration=1.107795635 podStartE2EDuration="1.86887319s" totalImagesPullingTime="761.077555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:33.867792312 +0000 UTC m=+975.770248510" watchObservedRunningTime="2026-09-18 14:00:33.86887319 +0000 UTC m=+975.771329373"
Sep 18 14:00:39.856402 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:39.856375    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:00:58.975278 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:58.975249    2569 generic.go:356] "Generic (PLEG): container finished" podID="94793ee8-4de8-486a-936d-3943505317d7" containerID="9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c" exitCode=0
Sep 18 14:00:58.975820 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:58.975332    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" event={"ID":"94793ee8-4de8-486a-936d-3943505317d7","Type":"ContainerDied","Data":"9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c"}
Sep 18 14:00:58.975913 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:58.975823    2569 scope.go:117] "RemoveContainer" containerID="9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c"
Sep 18 14:00:58.976894 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:58.976850    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:00:58.976998 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:58.976940    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:00:58.976998 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:58.976972    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:00:59.982953 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:00:59.982920    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" event={"ID":"94793ee8-4de8-486a-936d-3943505317d7","Type":"ContainerStarted","Data":"4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd"}
Sep 18 14:01:00.699389 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:00.699356    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:01:00.699582 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:00.699525    2569 patch_prober.go:28] interesting pod/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 14:01:00.699629 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:00.699572    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 14:01:01.699519 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:01.699489    2569 patch_prober.go:28] interesting pod/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 14:01:01.699894 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:01.699541    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 14:01:02.699436 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:02.699400    2569 patch_prober.go:28] interesting pod/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 14:01:02.699648 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:02.699452    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 14:01:03.699472 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:03.699438    2569 patch_prober.go:28] interesting pod/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 14:01:03.699718 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:03.699492    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 14:01:04.700034 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:04.699996    2569 patch_prober.go:28] interesting pod/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 14:01:04.700475 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:04.700091    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 14:01:05.703274 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:05.703246    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:01:05.703863 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:05.703526    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:01:05.704094 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:05.704062    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:01:15.794932 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:15.794903    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"]
Sep 18 14:01:15.795463 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:15.795167    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br" podUID="3a3a7012-dcf5-4128-ac00-4ca4ef1b9183" containerName="maas-api" containerID="cri-o://a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c" gracePeriod=30
Sep 18 14:01:16.043909 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.043888    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:01:16.062389 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.062297    2569 generic.go:356] "Generic (PLEG): container finished" podID="3a3a7012-dcf5-4128-ac00-4ca4ef1b9183" containerID="a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c" exitCode=0
Sep 18 14:01:16.062540 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.062479    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br" event={"ID":"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183","Type":"ContainerDied","Data":"a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c"}
Sep 18 14:01:16.062607 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.062538    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br" event={"ID":"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183","Type":"ContainerDied","Data":"50a43c53af1376815830809e908962dde45390bcf59d4016c44a80ed172f0123"}
Sep 18 14:01:16.062607 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.062488    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"
Sep 18 14:01:16.062713 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.062568    2569 scope.go:117] "RemoveContainer" containerID="a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c"
Sep 18 14:01:16.074412 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.074391    2569 scope.go:117] "RemoveContainer" containerID="a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c"
Sep 18 14:01:16.074767 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:01:16.074739    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c\": container with ID starting with a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c not found: ID does not exist" containerID="a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c"
Sep 18 14:01:16.074849 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.074779    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c"} err="failed to get container status \"a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c\": rpc error: code = NotFound desc = could not find container \"a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c\": container with ID starting with a4bfd54fd9fa9484b7817a67250041587dc2ea148fdd6328e4eacab15d1d1d2c not found: ID does not exist"
Sep 18 14:01:16.112256 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.112237    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-metrics-certs\") pod \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") "
Sep 18 14:01:16.112373 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.112295    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-maas-api-tls\") pod \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") "
Sep 18 14:01:16.112373 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.112340    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhznb\" (UniqueName: \"kubernetes.io/projected/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-kube-api-access-jhznb\") pod \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") "
Sep 18 14:01:16.112457 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.112371    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-openshift-service-ca\") pod \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\" (UID: \"3a3a7012-dcf5-4128-ac00-4ca4ef1b9183\") "
Sep 18 14:01:16.112811 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.112782    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a3a7012-dcf5-4128-ac00-4ca4ef1b9183" (UID: "3a3a7012-dcf5-4128-ac00-4ca4ef1b9183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:16.114674 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.114644    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a3a7012-dcf5-4128-ac00-4ca4ef1b9183" (UID: "3a3a7012-dcf5-4128-ac00-4ca4ef1b9183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:16.114674 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.114655    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-kube-api-access-jhznb" (OuterVolumeSpecName: "kube-api-access-jhznb") pod "3a3a7012-dcf5-4128-ac00-4ca4ef1b9183" (UID: "3a3a7012-dcf5-4128-ac00-4ca4ef1b9183"). InnerVolumeSpecName "kube-api-access-jhznb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:16.114835 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.114791    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3a3a7012-dcf5-4128-ac00-4ca4ef1b9183" (UID: "3a3a7012-dcf5-4128-ac00-4ca4ef1b9183"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:16.213609 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.213569    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhznb\" (UniqueName: \"kubernetes.io/projected/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-kube-api-access-jhznb\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:16.213609 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.213601    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:16.213609 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.213616    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:16.213905 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.213628    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:16.386963 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.386939    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"]
Sep 18 14:01:16.390899 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.390873    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bc6110-548bf68fbf-4w9br"]
Sep 18 14:01:16.486592 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:16.486556    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a3a7012-dcf5-4128-ac00-4ca4ef1b9183" path="/var/lib/kubelet/pods/3a3a7012-dcf5-4128-ac00-4ca4ef1b9183/volumes"
Sep 18 14:01:23.230525 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:23.230494    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"]
Sep 18 14:01:23.230956 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:23.230690    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy" containerID="cri-o://4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd" gracePeriod=30
Sep 18 14:01:28.485004 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.484977    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:01:28.626827 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.626734    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-credential-socket\") pod \"94793ee8-4de8-486a-936d-3943505317d7\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") "
Sep 18 14:01:28.626973 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.626855    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94793ee8-4de8-486a-936d-3943505317d7-istiod-ca-cert\") pod \"94793ee8-4de8-486a-936d-3943505317d7\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") "
Sep 18 14:01:28.626973 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.626888    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-socket\") pod \"94793ee8-4de8-486a-936d-3943505317d7\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") "
Sep 18 14:01:28.626973 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.626923    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-certs\") pod \"94793ee8-4de8-486a-936d-3943505317d7\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") "
Sep 18 14:01:28.627144 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.626973    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9v46\" (UniqueName: \"kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-kube-api-access-x9v46\") pod \"94793ee8-4de8-486a-936d-3943505317d7\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") "
Sep 18 14:01:28.627144 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.626996    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-envoy\") pod \"94793ee8-4de8-486a-936d-3943505317d7\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") "
Sep 18 14:01:28.627144 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627023    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94793ee8-4de8-486a-936d-3943505317d7-istio-podinfo\") pod \"94793ee8-4de8-486a-936d-3943505317d7\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") "
Sep 18 14:01:28.627144 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627033    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "94793ee8-4de8-486a-936d-3943505317d7" (UID: "94793ee8-4de8-486a-936d-3943505317d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:01:28.627144 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627098    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-data\") pod \"94793ee8-4de8-486a-936d-3943505317d7\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") "
Sep 18 14:01:28.627144 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627129    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-istio-token\") pod \"94793ee8-4de8-486a-936d-3943505317d7\" (UID: \"94793ee8-4de8-486a-936d-3943505317d7\") "
Sep 18 14:01:28.627456 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627182    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "94793ee8-4de8-486a-936d-3943505317d7" (UID: "94793ee8-4de8-486a-936d-3943505317d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:01:28.627456 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627302    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "94793ee8-4de8-486a-936d-3943505317d7" (UID: "94793ee8-4de8-486a-936d-3943505317d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:01:28.627456 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627311    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94793ee8-4de8-486a-936d-3943505317d7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "94793ee8-4de8-486a-936d-3943505317d7" (UID: "94793ee8-4de8-486a-936d-3943505317d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:28.627608 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627505    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-credential-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:28.627608 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627525    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94793ee8-4de8-486a-936d-3943505317d7-istiod-ca-cert\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:28.627608 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627539    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:28.627765 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627686    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-workload-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:28.627765 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.627632    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-data" (OuterVolumeSpecName: "istio-data") pod "94793ee8-4de8-486a-936d-3943505317d7" (UID: "94793ee8-4de8-486a-936d-3943505317d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:01:28.629491 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.629470    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94793ee8-4de8-486a-936d-3943505317d7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "94793ee8-4de8-486a-936d-3943505317d7" (UID: "94793ee8-4de8-486a-936d-3943505317d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:01:28.629594 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.629546    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-kube-api-access-x9v46" (OuterVolumeSpecName: "kube-api-access-x9v46") pod "94793ee8-4de8-486a-936d-3943505317d7" (UID: "94793ee8-4de8-486a-936d-3943505317d7"). InnerVolumeSpecName "kube-api-access-x9v46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:28.629830 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.629804    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "94793ee8-4de8-486a-936d-3943505317d7" (UID: "94793ee8-4de8-486a-936d-3943505317d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:01:28.629890 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.629813    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-istio-token" (OuterVolumeSpecName: "istio-token") pod "94793ee8-4de8-486a-936d-3943505317d7" (UID: "94793ee8-4de8-486a-936d-3943505317d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:28.728537 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.728493    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9v46\" (UniqueName: \"kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-kube-api-access-x9v46\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:28.728537 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.728532    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-envoy\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:28.728537 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.728541    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94793ee8-4de8-486a-936d-3943505317d7-istio-podinfo\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:28.728537 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.728550    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94793ee8-4de8-486a-936d-3943505317d7-istio-data\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:28.728790 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:28.728558    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94793ee8-4de8-486a-936d-3943505317d7-istio-token\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:29.134712 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.134677    2569 generic.go:356] "Generic (PLEG): container finished" podID="94793ee8-4de8-486a-936d-3943505317d7" containerID="4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd" exitCode=0
Sep 18 14:01:29.134894 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.134721    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"
Sep 18 14:01:29.134894 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.134757    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" event={"ID":"94793ee8-4de8-486a-936d-3943505317d7","Type":"ContainerDied","Data":"4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd"}
Sep 18 14:01:29.134894 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.134809    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq" event={"ID":"94793ee8-4de8-486a-936d-3943505317d7","Type":"ContainerDied","Data":"bee3de5202bf18599a516e5f4de25d1eb332c4c4805aeaed2b0eea9b6755e65e"}
Sep 18 14:01:29.134894 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.134832    2569 scope.go:117] "RemoveContainer" containerID="4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd"
Sep 18 14:01:29.144807 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.144790    2569 scope.go:117] "RemoveContainer" containerID="9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c"
Sep 18 14:01:29.152885 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.152814    2569 scope.go:117] "RemoveContainer" containerID="4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd"
Sep 18 14:01:29.153125 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:01:29.153107    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd\": container with ID starting with 4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd not found: ID does not exist" containerID="4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd"
Sep 18 14:01:29.153198 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.153133    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd"} err="failed to get container status \"4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd\": rpc error: code = NotFound desc = could not find container \"4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd\": container with ID starting with 4323804a35015c225a7b532ba6482839e698e73518502d61ad1f3bda17530ddd not found: ID does not exist"
Sep 18 14:01:29.153198 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.153148    2569 scope.go:117] "RemoveContainer" containerID="9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c"
Sep 18 14:01:29.153393 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:01:29.153378    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c\": container with ID starting with 9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c not found: ID does not exist" containerID="9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c"
Sep 18 14:01:29.153434 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.153397    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c"} err="failed to get container status \"9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c\": rpc error: code = NotFound desc = could not find container \"9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c\": container with ID starting with 9f387e3a4c7e5cb35f12591b8691ebaa085e9728789c52f489ab41d7d7969a8c not found: ID does not exist"
Sep 18 14:01:29.163376 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.163354    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"]
Sep 18 14:01:29.165591 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:29.165572    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bc6110-openshift-default-846bb576b4-hvcnq"]
Sep 18 14:01:30.485689 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:30.485660    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94793ee8-4de8-486a-936d-3943505317d7" path="/var/lib/kubelet/pods/94793ee8-4de8-486a-936d-3943505317d7/volumes"
Sep 18 14:01:38.435460 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.435430    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"]
Sep 18 14:01:38.435847 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.435832    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy"
Sep 18 14:01:38.435900 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.435851    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy"
Sep 18 14:01:38.435900 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.435866    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy"
Sep 18 14:01:38.435900 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.435871    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy"
Sep 18 14:01:38.435900 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.435878    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3a7012-dcf5-4128-ac00-4ca4ef1b9183" containerName="maas-api"
Sep 18 14:01:38.435900 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.435883    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3a7012-dcf5-4128-ac00-4ca4ef1b9183" containerName="maas-api"
Sep 18 14:01:38.436055 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.435967    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy"
Sep 18 14:01:38.436055 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.435974    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3a7012-dcf5-4128-ac00-4ca4ef1b9183" containerName="maas-api"
Sep 18 14:01:38.438995 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.438971    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.442764 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.442741    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-e46fbf-openshift-default-dockercfg-2hn42\""
Sep 18 14:01:38.455356 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.455333    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"]
Sep 18 14:01:38.617488 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.617450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgc8b\" (UniqueName: \"kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-kube-api-access-lgc8b\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.617665 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.617514    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-certs\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.617665 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.617562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-data\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.617665 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.617604    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c0ae47-d6db-4759-96e8-1c803ddedf94-istiod-ca-cert\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.617665 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.617648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-credential-socket\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.617890 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.617670    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-podinfo\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.617890 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.617705    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-envoy\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.617890 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.617808    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-socket\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.617890 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.617852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-token\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719029 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.718994    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-credential-socket\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719233 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719035    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-podinfo\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719233 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-envoy\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719233 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719143    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-socket\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719233 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-token\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719233 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719207    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgc8b\" (UniqueName: \"kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-kube-api-access-lgc8b\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719507 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-certs\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719507 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719301    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-data\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719507 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c0ae47-d6db-4759-96e8-1c803ddedf94-istiod-ca-cert\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719727 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-credential-socket\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719727 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719595    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-socket\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719727 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719624    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-certs\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.719727 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.719716    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-data\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.720108 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.720057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c0ae47-d6db-4759-96e8-1c803ddedf94-istiod-ca-cert\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.721819 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.721799    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-podinfo\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.721924 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.721869    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-envoy\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.727633 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.727612    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-token\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.727763 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.727745    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgc8b\" (UniqueName: \"kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-kube-api-access-lgc8b\") pod \"e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") " pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:38.750407 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:38.750383    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:39.025448 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:39.025357    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"]
Sep 18 14:01:39.027794 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:01:39.027766    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c0ae47_d6db_4759_96e8_1c803ddedf94.slice/crio-5b101c5b36ac6d24570ea79dc94d57c70c7679f5e857800f976a806ae4b6a3bc WatchSource:0}: Error finding container 5b101c5b36ac6d24570ea79dc94d57c70c7679f5e857800f976a806ae4b6a3bc: Status 404 returned error can't find the container with id 5b101c5b36ac6d24570ea79dc94d57c70c7679f5e857800f976a806ae4b6a3bc
Sep 18 14:01:39.030040 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:39.030013    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:01:39.030132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:39.030092    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:01:39.030132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:39.030119    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:01:39.187273 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:39.187236    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" event={"ID":"95c0ae47-d6db-4759-96e8-1c803ddedf94","Type":"ContainerStarted","Data":"b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816"}
Sep 18 14:01:39.187472 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:39.187292    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" event={"ID":"95c0ae47-d6db-4759-96e8-1c803ddedf94","Type":"ContainerStarted","Data":"5b101c5b36ac6d24570ea79dc94d57c70c7679f5e857800f976a806ae4b6a3bc"}
Sep 18 14:01:39.215669 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:39.215612    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" podStartSLOduration=1.215591742 podStartE2EDuration="1.215591742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:01:39.214539053 +0000 UTC m=+1041.116995250" watchObservedRunningTime="2026-09-18 14:01:39.215591742 +0000 UTC m=+1041.118047932"
Sep 18 14:01:39.751204 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:39.751167    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:40.756578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:40.756546    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:41.198504 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:41.198456    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:41.199653 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:41.199629    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:01:42.915713 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:42.915682    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"]
Sep 18 14:01:42.916193 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:42.915908    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk" podUID="f7b0e5ab-032a-466b-b537-e5c13dff5f1e" containerName="payload-processing" containerID="cri-o://4b0153b15159bb4a35610a9605134dcb3e92e1164075c408a0b28e936873c94f" gracePeriod=30
Sep 18 14:01:42.921307 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:42.921280    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"]
Sep 18 14:01:42.921566 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:42.921514    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89" podUID="b16e2b5a-cf53-430c-81a4-0de5919ab4fb" containerName="payload-pre-processing" containerID="cri-o://faee60d71394553a9007b51cf4f2d2b96241771ff317dd345b69135c9d072b82" gracePeriod=30
Sep 18 14:01:43.211175 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.211146    2569 generic.go:356] "Generic (PLEG): container finished" podID="b16e2b5a-cf53-430c-81a4-0de5919ab4fb" containerID="faee60d71394553a9007b51cf4f2d2b96241771ff317dd345b69135c9d072b82" exitCode=0
Sep 18 14:01:43.211339 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.211219    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89" event={"ID":"b16e2b5a-cf53-430c-81a4-0de5919ab4fb","Type":"ContainerDied","Data":"faee60d71394553a9007b51cf4f2d2b96241771ff317dd345b69135c9d072b82"}
Sep 18 14:01:43.212763 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.212742    2569 generic.go:356] "Generic (PLEG): container finished" podID="f7b0e5ab-032a-466b-b537-e5c13dff5f1e" containerID="4b0153b15159bb4a35610a9605134dcb3e92e1164075c408a0b28e936873c94f" exitCode=0
Sep 18 14:01:43.212873 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.212784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk" event={"ID":"f7b0e5ab-032a-466b-b537-e5c13dff5f1e","Type":"ContainerDied","Data":"4b0153b15159bb4a35610a9605134dcb3e92e1164075c408a0b28e936873c94f"}
Sep 18 14:01:43.212873 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.212826    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk" event={"ID":"f7b0e5ab-032a-466b-b537-e5c13dff5f1e","Type":"ContainerDied","Data":"0fada1f34aa5fc8017ccea3e01756c9e6040146408f22af5efd46ee9fa7deb2f"}
Sep 18 14:01:43.212873 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.212848    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fada1f34aa5fc8017ccea3e01756c9e6040146408f22af5efd46ee9fa7deb2f"
Sep 18 14:01:43.229260 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.229235    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 14:01:43.232356 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.232338    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 14:01:43.365692 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.365657    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdd27\" (UniqueName: \"kubernetes.io/projected/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-kube-api-access-bdd27\") pod \"b16e2b5a-cf53-430c-81a4-0de5919ab4fb\" (UID: \"b16e2b5a-cf53-430c-81a4-0de5919ab4fb\") "
Sep 18 14:01:43.367782 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.367755    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7j2g\" (UniqueName: \"kubernetes.io/projected/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-kube-api-access-k7j2g\") pod \"f7b0e5ab-032a-466b-b537-e5c13dff5f1e\" (UID: \"f7b0e5ab-032a-466b-b537-e5c13dff5f1e\") "
Sep 18 14:01:43.368063 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.368042    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-plugins-config-volume\") pod \"b16e2b5a-cf53-430c-81a4-0de5919ab4fb\" (UID: \"b16e2b5a-cf53-430c-81a4-0de5919ab4fb\") "
Sep 18 14:01:43.368244 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.368217    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-plugins-config-volume\") pod \"f7b0e5ab-032a-466b-b537-e5c13dff5f1e\" (UID: \"f7b0e5ab-032a-466b-b537-e5c13dff5f1e\") "
Sep 18 14:01:43.369369 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.369324    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7b0e5ab-032a-466b-b537-e5c13dff5f1e" (UID: "f7b0e5ab-032a-466b-b537-e5c13dff5f1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:43.369490 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.369364    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b16e2b5a-cf53-430c-81a4-0de5919ab4fb" (UID: "b16e2b5a-cf53-430c-81a4-0de5919ab4fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:43.369558 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.369518    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-kube-api-access-bdd27" (OuterVolumeSpecName: "kube-api-access-bdd27") pod "b16e2b5a-cf53-430c-81a4-0de5919ab4fb" (UID: "b16e2b5a-cf53-430c-81a4-0de5919ab4fb"). InnerVolumeSpecName "kube-api-access-bdd27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:43.370925 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.370905    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-kube-api-access-k7j2g" (OuterVolumeSpecName: "kube-api-access-k7j2g") pod "f7b0e5ab-032a-466b-b537-e5c13dff5f1e" (UID: "f7b0e5ab-032a-466b-b537-e5c13dff5f1e"). InnerVolumeSpecName "kube-api-access-k7j2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:43.469664 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.469629    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdd27\" (UniqueName: \"kubernetes.io/projected/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-kube-api-access-bdd27\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:43.469664 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.469659    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7j2g\" (UniqueName: \"kubernetes.io/projected/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-kube-api-access-k7j2g\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:43.469664 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.469670    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e2b5a-cf53-430c-81a4-0de5919ab4fb-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:43.469930 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:43.469678    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0e5ab-032a-466b-b537-e5c13dff5f1e-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:01:44.219547 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.219516    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"
Sep 18 14:01:44.220372 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.219547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89" event={"ID":"b16e2b5a-cf53-430c-81a4-0de5919ab4fb","Type":"ContainerDied","Data":"f0dd5e09051f69b7193619e4882b352d3d68213fd5ff070a019d9353a1bbd4f1"}
Sep 18 14:01:44.220372 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.219602    2569 scope.go:117] "RemoveContainer" containerID="faee60d71394553a9007b51cf4f2d2b96241771ff317dd345b69135c9d072b82"
Sep 18 14:01:44.220372 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.219734    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"
Sep 18 14:01:44.247187 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.247153    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"]
Sep 18 14:01:44.250597 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.250566    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0aac26-9b87f8bcc-6f7nk"]
Sep 18 14:01:44.264004 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.263974    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"]
Sep 18 14:01:44.269205 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.269179    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0aac26-79cfc5f646-bjm89"]
Sep 18 14:01:44.486393 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.486306    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b16e2b5a-cf53-430c-81a4-0de5919ab4fb" path="/var/lib/kubelet/pods/b16e2b5a-cf53-430c-81a4-0de5919ab4fb/volumes"
Sep 18 14:01:44.486684 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.486671    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b0e5ab-032a-466b-b537-e5c13dff5f1e" path="/var/lib/kubelet/pods/f7b0e5ab-032a-466b-b537-e5c13dff5f1e/volumes"
Sep 18 14:01:44.708581 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.708548    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"]
Sep 18 14:01:44.709126 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.709112    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b16e2b5a-cf53-430c-81a4-0de5919ab4fb" containerName="payload-pre-processing"
Sep 18 14:01:44.709172 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.709130    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b16e2b5a-cf53-430c-81a4-0de5919ab4fb" containerName="payload-pre-processing"
Sep 18 14:01:44.709172 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.709166    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b0e5ab-032a-466b-b537-e5c13dff5f1e" containerName="payload-processing"
Sep 18 14:01:44.709249 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.709174    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b0e5ab-032a-466b-b537-e5c13dff5f1e" containerName="payload-processing"
Sep 18 14:01:44.709282 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.709261    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="94793ee8-4de8-486a-936d-3943505317d7" containerName="istio-proxy"
Sep 18 14:01:44.709282 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.709274    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b0e5ab-032a-466b-b537-e5c13dff5f1e" containerName="payload-processing"
Sep 18 14:01:44.709342 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.709287    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b16e2b5a-cf53-430c-81a4-0de5919ab4fb" containerName="payload-pre-processing"
Sep 18 14:01:44.712520 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.712493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.718504 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.718482    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-e46fbf-serving-cert\""
Sep 18 14:01:44.727578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.727553    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"]
Sep 18 14:01:44.781295 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.781205    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.781295 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.781254    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.781295 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.781290    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngfl\" (UniqueName: \"kubernetes.io/projected/998e310e-0a23-49a3-bc01-f5c827b20c08-kube-api-access-zngfl\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.781580 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.781392    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/998e310e-0a23-49a3-bc01-f5c827b20c08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.882418 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.882388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/998e310e-0a23-49a3-bc01-f5c827b20c08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.882610 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.882474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.882610 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.882504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.882610 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.882531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngfl\" (UniqueName: \"kubernetes.io/projected/998e310e-0a23-49a3-bc01-f5c827b20c08-kube-api-access-zngfl\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.882787 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:01:44.882653    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-serving-cert: secret "maas-api-e2e-shared-a-w2-e46fbf-serving-cert" not found
Sep 18 14:01:44.882787 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:01:44.882738    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-maas-api-tls podName:998e310e-0a23-49a3-bc01-f5c827b20c08 nodeName:}" failed. No retries permitted until 2026-09-18 14:01:45.382714451 +0000 UTC m=+1047.285170634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-maas-api-tls") pod "maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2" (UID: "998e310e-0a23-49a3-bc01-f5c827b20c08") : secret "maas-api-e2e-shared-a-w2-e46fbf-serving-cert" not found
Sep 18 14:01:44.883135 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.883117    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/998e310e-0a23-49a3-bc01-f5c827b20c08-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.885080 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.885044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:44.893781 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:44.893751    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngfl\" (UniqueName: \"kubernetes.io/projected/998e310e-0a23-49a3-bc01-f5c827b20c08-kube-api-access-zngfl\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:45.324456 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.324423    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"]
Sep 18 14:01:45.326694 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.326675    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:01:45.330557 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.330537    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e46fbf-dockercfg-66pfs\""
Sep 18 14:01:45.330676 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.330594    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e46fbf\""
Sep 18 14:01:45.339692 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.339671    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"]
Sep 18 14:01:45.385692 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.385667    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:45.385804 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.385746    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dptbv\" (UniqueName: \"kubernetes.io/projected/7c48bddc-6e45-430f-97be-753cf5efd021-kube-api-access-dptbv\") pod \"payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq\" (UID: \"7c48bddc-6e45-430f-97be-753cf5efd021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:01:45.385804 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.385779    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c48bddc-6e45-430f-97be-753cf5efd021-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq\" (UID: \"7c48bddc-6e45-430f-97be-753cf5efd021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:01:45.388177 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.388154    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:45.487190 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.487151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dptbv\" (UniqueName: \"kubernetes.io/projected/7c48bddc-6e45-430f-97be-753cf5efd021-kube-api-access-dptbv\") pod \"payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq\" (UID: \"7c48bddc-6e45-430f-97be-753cf5efd021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:01:45.487402 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.487212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c48bddc-6e45-430f-97be-753cf5efd021-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq\" (UID: \"7c48bddc-6e45-430f-97be-753cf5efd021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:01:45.488171 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.488147    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c48bddc-6e45-430f-97be-753cf5efd021-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq\" (UID: \"7c48bddc-6e45-430f-97be-753cf5efd021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:01:45.495908 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.495881    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dptbv\" (UniqueName: \"kubernetes.io/projected/7c48bddc-6e45-430f-97be-753cf5efd021-kube-api-access-dptbv\") pod \"payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq\" (UID: \"7c48bddc-6e45-430f-97be-753cf5efd021\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:01:45.623482 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.623407    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:45.637547 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.637515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:01:45.904501 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.904215    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"]
Sep 18 14:01:45.907466 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:01:45.907413    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998e310e_0a23_49a3_bc01_f5c827b20c08.slice/crio-e5d663195762b68683bfc9c4ac77ef751bac14b8bed0fad413967d1d651dafc3 WatchSource:0}: Error finding container e5d663195762b68683bfc9c4ac77ef751bac14b8bed0fad413967d1d651dafc3: Status 404 returned error can't find the container with id e5d663195762b68683bfc9c4ac77ef751bac14b8bed0fad413967d1d651dafc3
Sep 18 14:01:45.909459 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.909440    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:01:45.930058 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:45.930037    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"]
Sep 18 14:01:45.931738 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:01:45.931715    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c48bddc_6e45_430f_97be_753cf5efd021.slice/crio-914d4ed40a7ec0b52e549a5792f4a0e6b5ea69f870db51a84abb1bb7baaa6c21 WatchSource:0}: Error finding container 914d4ed40a7ec0b52e549a5792f4a0e6b5ea69f870db51a84abb1bb7baaa6c21: Status 404 returned error can't find the container with id 914d4ed40a7ec0b52e549a5792f4a0e6b5ea69f870db51a84abb1bb7baaa6c21
Sep 18 14:01:46.231563 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:46.231532    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq" event={"ID":"7c48bddc-6e45-430f-97be-753cf5efd021","Type":"ContainerStarted","Data":"317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114"}
Sep 18 14:01:46.231730 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:46.231582    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq" event={"ID":"7c48bddc-6e45-430f-97be-753cf5efd021","Type":"ContainerStarted","Data":"914d4ed40a7ec0b52e549a5792f4a0e6b5ea69f870db51a84abb1bb7baaa6c21"}
Sep 18 14:01:46.231730 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:46.231655    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:01:46.233029 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:46.232998    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2" event={"ID":"998e310e-0a23-49a3-bc01-f5c827b20c08","Type":"ContainerStarted","Data":"e5d663195762b68683bfc9c4ac77ef751bac14b8bed0fad413967d1d651dafc3"}
Sep 18 14:01:46.254105 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:46.254051    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq" podStartSLOduration=1.254039407 podStartE2EDuration="1.254039407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:01:46.25251464 +0000 UTC m=+1048.154970837" watchObservedRunningTime="2026-09-18 14:01:46.254039407 +0000 UTC m=+1048.156495646"
Sep 18 14:01:47.239477 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:47.239442    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2" event={"ID":"998e310e-0a23-49a3-bc01-f5c827b20c08","Type":"ContainerStarted","Data":"dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab"}
Sep 18 14:01:47.239932 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:47.239730    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:47.268028 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:47.267988    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2" podStartSLOduration=2.607600196 podStartE2EDuration="3.267973871s" totalImagesPullingTime="660.373675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:01:47.265516551 +0000 UTC m=+1049.167972766" watchObservedRunningTime="2026-09-18 14:01:47.267973871 +0000 UTC m=+1049.170430057"
Sep 18 14:01:53.251109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:53.251059    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:01:57.241531 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:01:57.241496    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:02:16.410127 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:16.410057    2569 generic.go:356] "Generic (PLEG): container finished" podID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerID="b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816" exitCode=0
Sep 18 14:02:16.410654 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:16.410133    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" event={"ID":"95c0ae47-d6db-4759-96e8-1c803ddedf94","Type":"ContainerDied","Data":"b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816"}
Sep 18 14:02:16.410654 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:16.410634    2569 scope.go:117] "RemoveContainer" containerID="b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816"
Sep 18 14:02:16.411849 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:16.411822    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:02:16.411910 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:16.411889    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:02:16.411947 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:16.411920    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:02:17.417731 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:17.417703    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" event={"ID":"95c0ae47-d6db-4759-96e8-1c803ddedf94","Type":"ContainerStarted","Data":"cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc"}
Sep 18 14:02:17.751463 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:17.751427    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:02:17.751923 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:17.751778    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:02:17.752148 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:17.752122    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:02:18.751972 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:18.751893    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:02:18.752568 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:18.752537    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:02:19.071087 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.071030    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"]
Sep 18 14:02:19.081954 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.081752    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"]
Sep 18 14:02:19.081954 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.081867    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.084841 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.084795    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-55bd4d-serving-cert\""
Sep 18 14:02:19.128217 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.128180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhphp\" (UniqueName: \"kubernetes.io/projected/b6941922-7692-4ac6-b5ea-fe1139735fb9-kube-api-access-rhphp\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.128420 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.128240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.128420 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.128295    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.128420 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.128398    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6941922-7692-4ac6-b5ea-fe1139735fb9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.229630 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.229594    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhphp\" (UniqueName: \"kubernetes.io/projected/b6941922-7692-4ac6-b5ea-fe1139735fb9-kube-api-access-rhphp\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.229849 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.229679    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.229849 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.229710    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.229849 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.229749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6941922-7692-4ac6-b5ea-fe1139735fb9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.230487 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.230453    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6941922-7692-4ac6-b5ea-fe1139735fb9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.232472 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.232438    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.232472 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.232466    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.249097 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.242131    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhphp\" (UniqueName: \"kubernetes.io/projected/b6941922-7692-4ac6-b5ea-fe1139735fb9-kube-api-access-rhphp\") pod \"maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.393887 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.393801    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:19.696767 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.696742    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"]
Sep 18 14:02:19.698759 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:02:19.698727    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6941922_7692_4ac6_b5ea_fe1139735fb9.slice/crio-45f6d115b1d23a4b3a198b2b8b64cb8485587ee9a735b9c9e2143cc4ae6936dc WatchSource:0}: Error finding container 45f6d115b1d23a4b3a198b2b8b64cb8485587ee9a735b9c9e2143cc4ae6936dc: Status 404 returned error can't find the container with id 45f6d115b1d23a4b3a198b2b8b64cb8485587ee9a735b9c9e2143cc4ae6936dc
Sep 18 14:02:19.751528 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.751490    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:02:19.752192 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.751777    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:02:19.777427 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.777400    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"]
Sep 18 14:02:19.780474 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.780454    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:02:19.783428 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.783411    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-55bd4d\""
Sep 18 14:02:19.783547 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.783509    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-55bd4d-dockercfg-jcf4q\""
Sep 18 14:02:19.793090 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.793042    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"]
Sep 18 14:02:19.836403 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.836375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937777ed-724d-4b69-865d-0e9e64d05aaf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs\" (UID: \"937777ed-724d-4b69-865d-0e9e64d05aaf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:02:19.836534 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.836442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjmnm\" (UniqueName: \"kubernetes.io/projected/937777ed-724d-4b69-865d-0e9e64d05aaf-kube-api-access-pjmnm\") pod \"payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs\" (UID: \"937777ed-724d-4b69-865d-0e9e64d05aaf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:02:19.939679 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.939637    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjmnm\" (UniqueName: \"kubernetes.io/projected/937777ed-724d-4b69-865d-0e9e64d05aaf-kube-api-access-pjmnm\") pod \"payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs\" (UID: \"937777ed-724d-4b69-865d-0e9e64d05aaf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:02:19.939890 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.939784    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937777ed-724d-4b69-865d-0e9e64d05aaf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs\" (UID: \"937777ed-724d-4b69-865d-0e9e64d05aaf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:02:19.940755 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.940723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937777ed-724d-4b69-865d-0e9e64d05aaf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs\" (UID: \"937777ed-724d-4b69-865d-0e9e64d05aaf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:02:19.955006 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:19.954933    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjmnm\" (UniqueName: \"kubernetes.io/projected/937777ed-724d-4b69-865d-0e9e64d05aaf-kube-api-access-pjmnm\") pod \"payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs\" (UID: \"937777ed-724d-4b69-865d-0e9e64d05aaf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:02:20.092343 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:20.092316    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:02:20.390823 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:20.390802    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"]
Sep 18 14:02:20.391603 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:02:20.391578    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937777ed_724d_4b69_865d_0e9e64d05aaf.slice/crio-255debc167772a1307e1209886a38b5f64fd4f047be3861000660958e02ab65b WatchSource:0}: Error finding container 255debc167772a1307e1209886a38b5f64fd4f047be3861000660958e02ab65b: Status 404 returned error can't find the container with id 255debc167772a1307e1209886a38b5f64fd4f047be3861000660958e02ab65b
Sep 18 14:02:20.441095 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:20.437199    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc" event={"ID":"b6941922-7692-4ac6-b5ea-fe1139735fb9","Type":"ContainerStarted","Data":"0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364"}
Sep 18 14:02:20.441095 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:20.437259    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc" event={"ID":"b6941922-7692-4ac6-b5ea-fe1139735fb9","Type":"ContainerStarted","Data":"45f6d115b1d23a4b3a198b2b8b64cb8485587ee9a735b9c9e2143cc4ae6936dc"}
Sep 18 14:02:20.441095 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:20.437304    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:20.446105 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:20.444734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs" event={"ID":"937777ed-724d-4b69-865d-0e9e64d05aaf","Type":"ContainerStarted","Data":"255debc167772a1307e1209886a38b5f64fd4f047be3861000660958e02ab65b"}
Sep 18 14:02:20.751109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:20.751048    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:02:20.751414 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:20.751375    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:02:21.457420 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:21.457383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs" event={"ID":"937777ed-724d-4b69-865d-0e9e64d05aaf","Type":"ContainerStarted","Data":"08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5"}
Sep 18 14:02:21.458810 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:21.458788    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:02:21.478508 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:21.478447    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc" podStartSLOduration=2.185294501 podStartE2EDuration="2.478428368s" totalImagesPullingTime="293.133867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:20.465148069 +0000 UTC m=+1082.367604255" watchObservedRunningTime="2026-09-18 14:02:21.478428368 +0000 UTC m=+1083.380884555"
Sep 18 14:02:21.478953 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:21.478908    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs" podStartSLOduration=2.478895667 podStartE2EDuration="2.478895667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:21.47722336 +0000 UTC m=+1083.379679545" watchObservedRunningTime="2026-09-18 14:02:21.478895667 +0000 UTC m=+1083.381351858"
Sep 18 14:02:21.751942 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:21.751836    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:02:21.751942 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:21.751907    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:02:22.751630 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:22.751583    2569 patch_prober.go:28] interesting pod/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 18 14:02:22.752187 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:22.751660    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 18 14:02:23.755715 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:23.755685    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:02:23.756222 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:23.756108    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:02:23.756679 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:23.756663    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:02:26.398452 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.398420    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"]
Sep 18 14:02:26.399221 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.399190    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp" podUID="e5e953bc-4256-4fd6-88d0-244131c1ccc6" containerName="maas-api" containerID="cri-o://96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2" gracePeriod=30
Sep 18 14:02:26.464292 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.464260    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:02:26.664921 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.664873    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 14:02:26.719394 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.719360    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcs6p\" (UniqueName: \"kubernetes.io/projected/e5e953bc-4256-4fd6-88d0-244131c1ccc6-kube-api-access-kcs6p\") pod \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") "
Sep 18 14:02:26.719560 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.719473    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-metrics-certs\") pod \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") "
Sep 18 14:02:26.719560 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.719507    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e953bc-4256-4fd6-88d0-244131c1ccc6-openshift-service-ca\") pod \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") "
Sep 18 14:02:26.719680 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.719566    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-maas-api-tls\") pod \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\" (UID: \"e5e953bc-4256-4fd6-88d0-244131c1ccc6\") "
Sep 18 14:02:26.720463 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.720427    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e953bc-4256-4fd6-88d0-244131c1ccc6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5e953bc-4256-4fd6-88d0-244131c1ccc6" (UID: "e5e953bc-4256-4fd6-88d0-244131c1ccc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:02:26.721854 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.721829    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e953bc-4256-4fd6-88d0-244131c1ccc6-kube-api-access-kcs6p" (OuterVolumeSpecName: "kube-api-access-kcs6p") pod "e5e953bc-4256-4fd6-88d0-244131c1ccc6" (UID: "e5e953bc-4256-4fd6-88d0-244131c1ccc6"). InnerVolumeSpecName "kube-api-access-kcs6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:26.722409 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.722390    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5e953bc-4256-4fd6-88d0-244131c1ccc6" (UID: "e5e953bc-4256-4fd6-88d0-244131c1ccc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:02:26.722483 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.722418    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e5e953bc-4256-4fd6-88d0-244131c1ccc6" (UID: "e5e953bc-4256-4fd6-88d0-244131c1ccc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:02:26.820932 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.820897    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:02:26.820932 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.820931    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e953bc-4256-4fd6-88d0-244131c1ccc6-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:02:26.821157 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.820944    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e953bc-4256-4fd6-88d0-244131c1ccc6-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:02:26.821157 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:26.820956    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcs6p\" (UniqueName: \"kubernetes.io/projected/e5e953bc-4256-4fd6-88d0-244131c1ccc6-kube-api-access-kcs6p\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:02:27.492884 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:27.492850    2569 generic.go:356] "Generic (PLEG): container finished" podID="e5e953bc-4256-4fd6-88d0-244131c1ccc6" containerID="96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2" exitCode=0
Sep 18 14:02:27.492884 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:27.492888    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp" event={"ID":"e5e953bc-4256-4fd6-88d0-244131c1ccc6","Type":"ContainerDied","Data":"96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2"}
Sep 18 14:02:27.493414 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:27.492895    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"
Sep 18 14:02:27.493414 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:27.492921    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp" event={"ID":"e5e953bc-4256-4fd6-88d0-244131c1ccc6","Type":"ContainerDied","Data":"3b41d8e60fef0318eaab3fb7a6ef447719f0e8395ef251281c322faf84247d74"}
Sep 18 14:02:27.493414 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:27.492942    2569 scope.go:117] "RemoveContainer" containerID="96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2"
Sep 18 14:02:27.502550 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:27.502524    2569 scope.go:117] "RemoveContainer" containerID="96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2"
Sep 18 14:02:27.502854 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:02:27.502833    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2\": container with ID starting with 96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2 not found: ID does not exist" containerID="96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2"
Sep 18 14:02:27.502931 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:27.502863    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2"} err="failed to get container status \"96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2\": rpc error: code = NotFound desc = could not find container \"96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2\": container with ID starting with 96fba504c0fc0fb2db0f1b29e0bcf53984199cce5fef47bab6e57348a53139f2 not found: ID does not exist"
Sep 18 14:02:27.516791 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:27.516770    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"]
Sep 18 14:02:27.520893 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:27.520518    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df2eb8-8f8955698-2lfnp"]
Sep 18 14:02:28.485980 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:28.485948    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e953bc-4256-4fd6-88d0-244131c1ccc6" path="/var/lib/kubelet/pods/e5e953bc-4256-4fd6-88d0-244131c1ccc6/volumes"
Sep 18 14:02:33.471578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:33.471546    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:02:46.902120 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:46.902081    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"]
Sep 18 14:02:46.902619 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:46.902602    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e953bc-4256-4fd6-88d0-244131c1ccc6" containerName="maas-api"
Sep 18 14:02:46.902679 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:46.902622    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e953bc-4256-4fd6-88d0-244131c1ccc6" containerName="maas-api"
Sep 18 14:02:46.902725 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:46.902715    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e953bc-4256-4fd6-88d0-244131c1ccc6" containerName="maas-api"
Sep 18 14:02:46.904757 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:46.904739    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:46.907529 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:46.907510    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bfd52075-serving-cert\""
Sep 18 14:02:46.917006 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:46.916985    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"]
Sep 18 14:02:47.013512 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.013474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7f4x\" (UniqueName: \"kubernetes.io/projected/38ae448b-ec07-4f7b-a997-49dfe36e22b5-kube-api-access-k7f4x\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.013687 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.013547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ae448b-ec07-4f7b-a997-49dfe36e22b5-openshift-service-ca\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.013687 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.013575    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-maas-api-tls\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.013687 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.013592    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-metrics-certs\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.114638 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.114611    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ae448b-ec07-4f7b-a997-49dfe36e22b5-openshift-service-ca\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.114814 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.114650    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-maas-api-tls\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.114814 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.114668    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-metrics-certs\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.114814 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.114716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7f4x\" (UniqueName: \"kubernetes.io/projected/38ae448b-ec07-4f7b-a997-49dfe36e22b5-kube-api-access-k7f4x\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.115004 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:02:47.114812    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-serving-cert: secret "maas-api-e2e-mt-bfd52075-serving-cert" not found
Sep 18 14:02:47.115004 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:02:47.114891    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-maas-api-tls podName:38ae448b-ec07-4f7b-a997-49dfe36e22b5 nodeName:}" failed. No retries permitted until 2026-09-18 14:02:47.614875194 +0000 UTC m=+1109.517331372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-maas-api-tls") pod "maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2" (UID: "38ae448b-ec07-4f7b-a997-49dfe36e22b5") : secret "maas-api-e2e-mt-bfd52075-serving-cert" not found
Sep 18 14:02:47.115447 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.115419    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ae448b-ec07-4f7b-a997-49dfe36e22b5-openshift-service-ca\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.117510 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.117488    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-metrics-certs\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.132354 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.132333    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7f4x\" (UniqueName: \"kubernetes.io/projected/38ae448b-ec07-4f7b-a997-49dfe36e22b5-kube-api-access-k7f4x\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.619886 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.619857    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-maas-api-tls\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.622469 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.622447    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-maas-api-tls\") pod \"maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:47.814696 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.814662    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:48.004093 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.995471    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"]
Sep 18 14:02:48.004093 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:47.995690    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9" podUID="2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79" containerName="maas-api" containerID="cri-o://ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65" gracePeriod=30
Sep 18 14:02:48.099186 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.099155    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"]
Sep 18 14:02:48.100218 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:02:48.100160    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ae448b_ec07_4f7b_a997_49dfe36e22b5.slice/crio-76fbd626307dee3230abb117d4a4a990b773d4301986ede7f64c26eaaaedcdf7 WatchSource:0}: Error finding container 76fbd626307dee3230abb117d4a4a990b773d4301986ede7f64c26eaaaedcdf7: Status 404 returned error can't find the container with id 76fbd626307dee3230abb117d4a4a990b773d4301986ede7f64c26eaaaedcdf7
Sep 18 14:02:48.280623 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.280598    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 14:02:48.330846 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.330812    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-maas-api-tls\") pod \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") "
Sep 18 14:02:48.331054 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.330871    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnb4n\" (UniqueName: \"kubernetes.io/projected/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-kube-api-access-cnb4n\") pod \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") "
Sep 18 14:02:48.331054 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.330920    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-openshift-service-ca\") pod \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") "
Sep 18 14:02:48.331054 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.330982    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-metrics-certs\") pod \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\" (UID: \"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79\") "
Sep 18 14:02:48.331371 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.331341    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79" (UID: "2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:02:48.333358 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.333304    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79" (UID: "2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:02:48.333358 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.333307    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-kube-api-access-cnb4n" (OuterVolumeSpecName: "kube-api-access-cnb4n") pod "2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79" (UID: "2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79"). InnerVolumeSpecName "kube-api-access-cnb4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:48.333529 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.333359    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79" (UID: "2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:02:48.432695 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.432658    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:02:48.432695 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.432688    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:02:48.432695 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.432697    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnb4n\" (UniqueName: \"kubernetes.io/projected/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-kube-api-access-cnb4n\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:02:48.432931 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.432705    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:02:48.612312 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.612281    2569 generic.go:356] "Generic (PLEG): container finished" podID="2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79" containerID="ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65" exitCode=0
Sep 18 14:02:48.612449 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.612322    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"
Sep 18 14:02:48.612449 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.612362    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9" event={"ID":"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79","Type":"ContainerDied","Data":"ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65"}
Sep 18 14:02:48.612449 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.612416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9" event={"ID":"2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79","Type":"ContainerDied","Data":"8d6b451e9c6c8017c25f8f715061f55fabeb376d1b1cb41dbfe47031f251293c"}
Sep 18 14:02:48.612449 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.612446    2569 scope.go:117] "RemoveContainer" containerID="ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65"
Sep 18 14:02:48.614602 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.614570    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2" event={"ID":"38ae448b-ec07-4f7b-a997-49dfe36e22b5","Type":"ContainerStarted","Data":"28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f"}
Sep 18 14:02:48.614723 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.614628    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2" event={"ID":"38ae448b-ec07-4f7b-a997-49dfe36e22b5","Type":"ContainerStarted","Data":"76fbd626307dee3230abb117d4a4a990b773d4301986ede7f64c26eaaaedcdf7"}
Sep 18 14:02:48.615222 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.615203    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:02:48.622016 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.621997    2569 scope.go:117] "RemoveContainer" containerID="ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65"
Sep 18 14:02:48.622338 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:02:48.622314    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65\": container with ID starting with ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65 not found: ID does not exist" containerID="ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65"
Sep 18 14:02:48.622448 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.622346    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65"} err="failed to get container status \"ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65\": rpc error: code = NotFound desc = could not find container \"ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65\": container with ID starting with ec187c7e38bce2216caff9fcbde8fc23fce73632762ab1cf2d49b8f512676b65 not found: ID does not exist"
Sep 18 14:02:48.645724 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.645686    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2" podStartSLOduration=2.225513951 podStartE2EDuration="2.645672489s" totalImagesPullingTime="420.158538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:48.643635783 +0000 UTC m=+1110.546091960" watchObservedRunningTime="2026-09-18 14:02:48.645672489 +0000 UTC m=+1110.548128675"
Sep 18 14:02:48.670167 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.670143    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"]
Sep 18 14:02:48.677582 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:48.677558    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b63f26-6c7b97c4cf-fwmz9"]
Sep 18 14:02:50.485847 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:50.485810    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79" path="/var/lib/kubelet/pods/2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79/volumes"
Sep 18 14:02:55.632606 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:02:55.632580    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:03:03.296764 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.296734    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"]
Sep 18 14:03:03.297259 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.297200    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79" containerName="maas-api"
Sep 18 14:03:03.297259 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.297213    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79" containerName="maas-api"
Sep 18 14:03:03.297365 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.297289    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f811cf4-a8ed-4d17-ab69-3bdc1eeb7d79" containerName="maas-api"
Sep 18 14:03:03.300762 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.300742    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.303455 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.303434    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-783872-serving-cert\""
Sep 18 14:03:03.315882 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.315858    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"]
Sep 18 14:03:03.365062 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.365033    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359731fd-a9c3-4aa2-b652-1574aaabf48a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.365242 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.365110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7psw\" (UniqueName: \"kubernetes.io/projected/359731fd-a9c3-4aa2-b652-1574aaabf48a-kube-api-access-t7psw\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.365242 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.365192    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.365242 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.365234    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.466041 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.466000    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.466279 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.466052    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.466279 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.466153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359731fd-a9c3-4aa2-b652-1574aaabf48a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.466279 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.466196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7psw\" (UniqueName: \"kubernetes.io/projected/359731fd-a9c3-4aa2-b652-1574aaabf48a-kube-api-access-t7psw\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.466279 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:03:03.466237    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-serving-cert: secret "maas-api-e2e-shared-a-w3-783872-serving-cert" not found
Sep 18 14:03:03.466498 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:03:03.466315    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-maas-api-tls podName:359731fd-a9c3-4aa2-b652-1574aaabf48a nodeName:}" failed. No retries permitted until 2026-09-18 14:03:03.966298131 +0000 UTC m=+1125.868754299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-maas-api-tls") pod "maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6" (UID: "359731fd-a9c3-4aa2-b652-1574aaabf48a") : secret "maas-api-e2e-shared-a-w3-783872-serving-cert" not found
Sep 18 14:03:03.466925 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.466903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359731fd-a9c3-4aa2-b652-1574aaabf48a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.468741 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.468720    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.476446 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.476426    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7psw\" (UniqueName: \"kubernetes.io/projected/359731fd-a9c3-4aa2-b652-1574aaabf48a-kube-api-access-t7psw\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.972490 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.972455    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:03.974919 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:03.974898    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:04.213642 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:04.213597    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:04.497577 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:04.497554    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"]
Sep 18 14:03:04.500369 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:03:04.500341    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359731fd_a9c3_4aa2_b652_1574aaabf48a.slice/crio-b350af84b405a7422ee333cb7ce491ecc122c49c23ce052827845cb9528266e3 WatchSource:0}: Error finding container b350af84b405a7422ee333cb7ce491ecc122c49c23ce052827845cb9528266e3: Status 404 returned error can't find the container with id b350af84b405a7422ee333cb7ce491ecc122c49c23ce052827845cb9528266e3
Sep 18 14:03:04.704158 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:04.704118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6" event={"ID":"359731fd-a9c3-4aa2-b652-1574aaabf48a","Type":"ContainerStarted","Data":"b350af84b405a7422ee333cb7ce491ecc122c49c23ce052827845cb9528266e3"}
Sep 18 14:03:05.711522 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:05.711482    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6" event={"ID":"359731fd-a9c3-4aa2-b652-1574aaabf48a","Type":"ContainerStarted","Data":"ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2"}
Sep 18 14:03:05.711973 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:05.711616    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:05.732761 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:05.732714    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6" podStartSLOduration=2.3122396 podStartE2EDuration="2.732698735s" totalImagesPullingTime="420.459135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:05.731137805 +0000 UTC m=+1127.633593992" watchObservedRunningTime="2026-09-18 14:03:05.732698735 +0000 UTC m=+1127.635154920"
Sep 18 14:03:11.723302 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:11.723266    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:03:39.156244 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.156204    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"]
Sep 18 14:03:39.161283 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.161266    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:03:39.164301 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.164284    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-1a4b6d-dockercfg-cdph2\""
Sep 18 14:03:39.164634 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.164620    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-1a4b6d\""
Sep 18 14:03:39.173046 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.173021    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"]
Sep 18 14:03:39.215777 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.215749    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dc29cc-a4f3-4646-9e96-28b245147efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg\" (UID: \"f1dc29cc-a4f3-4646-9e96-28b245147efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:03:39.215926 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.215784    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7p5x\" (UniqueName: \"kubernetes.io/projected/f1dc29cc-a4f3-4646-9e96-28b245147efd-kube-api-access-k7p5x\") pod \"payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg\" (UID: \"f1dc29cc-a4f3-4646-9e96-28b245147efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:03:39.316894 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.316857    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7p5x\" (UniqueName: \"kubernetes.io/projected/f1dc29cc-a4f3-4646-9e96-28b245147efd-kube-api-access-k7p5x\") pod \"payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg\" (UID: \"f1dc29cc-a4f3-4646-9e96-28b245147efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:03:39.317119 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.317004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dc29cc-a4f3-4646-9e96-28b245147efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg\" (UID: \"f1dc29cc-a4f3-4646-9e96-28b245147efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:03:39.317701 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.317676    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dc29cc-a4f3-4646-9e96-28b245147efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg\" (UID: \"f1dc29cc-a4f3-4646-9e96-28b245147efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:03:39.327272 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.327251    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7p5x\" (UniqueName: \"kubernetes.io/projected/f1dc29cc-a4f3-4646-9e96-28b245147efd-kube-api-access-k7p5x\") pod \"payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg\" (UID: \"f1dc29cc-a4f3-4646-9e96-28b245147efd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:03:39.472163 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.472128    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:03:39.748635 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.748611    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"]
Sep 18 14:03:39.750589 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:03:39.750562    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dc29cc_a4f3_4646_9e96_28b245147efd.slice/crio-02a543947a058b489bbd22280b7ed3b568189981dfba3d413ea3898445e24e67 WatchSource:0}: Error finding container 02a543947a058b489bbd22280b7ed3b568189981dfba3d413ea3898445e24e67: Status 404 returned error can't find the container with id 02a543947a058b489bbd22280b7ed3b568189981dfba3d413ea3898445e24e67
Sep 18 14:03:39.895601 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.895570    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg" event={"ID":"f1dc29cc-a4f3-4646-9e96-28b245147efd","Type":"ContainerStarted","Data":"b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526"}
Sep 18 14:03:39.895764 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.895618    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg" event={"ID":"f1dc29cc-a4f3-4646-9e96-28b245147efd","Type":"ContainerStarted","Data":"02a543947a058b489bbd22280b7ed3b568189981dfba3d413ea3898445e24e67"}
Sep 18 14:03:39.895764 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.895653    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:03:39.919860 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:39.919814    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg" podStartSLOduration=0.919799736 podStartE2EDuration="919.799736ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:39.917818855 +0000 UTC m=+1161.820275053" watchObservedRunningTime="2026-09-18 14:03:39.919799736 +0000 UTC m=+1161.822255986"
Sep 18 14:03:50.903022 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:03:50.902994    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:04:18.527587 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:04:18.527558    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 14:04:18.529620 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:04:18.529587    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 14:05:40.190460 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.190418    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"]
Sep 18 14:05:40.193878 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.193860    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.198141 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.198115    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"default-dockercfg-2qgqp\""
Sep 18 14:05:40.198435 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.198187    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"auth-test-model-1a4b6d-kserve-self-signed-certs\""
Sep 18 14:05:40.198435 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.198271    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"kube-root-ca.crt\""
Sep 18 14:05:40.198435 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.198277    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1a4b6d\"/\"openshift-service-ca.crt\""
Sep 18 14:05:40.214773 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.214750    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"]
Sep 18 14:05:40.326052 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.326018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-home\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.326255 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.326084    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-tmp-dir\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.326255 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.326116    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80282ae-09d3-42b0-9cfd-d1495199f99f-tls-certs\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.326255 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.326178    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-dshm\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.326255 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.326212    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-model-cache\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.326424 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.326260    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j49t\" (UniqueName: \"kubernetes.io/projected/e80282ae-09d3-42b0-9cfd-d1495199f99f-kube-api-access-5j49t\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.427646 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.427613    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80282ae-09d3-42b0-9cfd-d1495199f99f-tls-certs\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.427827 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.427658    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-dshm\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.427827 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.427681    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-model-cache\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.427827 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.427728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j49t\" (UniqueName: \"kubernetes.io/projected/e80282ae-09d3-42b0-9cfd-d1495199f99f-kube-api-access-5j49t\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.427984 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.427848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-home\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.427984 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.427882    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-tmp-dir\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.428228 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.428209    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-tmp-dir\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.428306 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.428210    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-model-cache\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.428306 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.428278    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-home\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.430047 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.430030    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-dshm\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.430451 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.430429    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80282ae-09d3-42b0-9cfd-d1495199f99f-tls-certs\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.436844 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.436827    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j49t\" (UniqueName: \"kubernetes.io/projected/e80282ae-09d3-42b0-9cfd-d1495199f99f-kube-api-access-5j49t\") pod \"auth-test-model-1a4b6d-kserve-965df48f9-7dbsx\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") " pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.508483 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.508445    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:40.792334 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:40.792311    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"]
Sep 18 14:05:40.794303 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:05:40.794273    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode80282ae_09d3_42b0_9cfd_d1495199f99f.slice/crio-5abf01f0ebaca2277c488ec2bce187e84f592c6b12b4a38c968cfcb7e4cf10ee WatchSource:0}: Error finding container 5abf01f0ebaca2277c488ec2bce187e84f592c6b12b4a38c968cfcb7e4cf10ee: Status 404 returned error can't find the container with id 5abf01f0ebaca2277c488ec2bce187e84f592c6b12b4a38c968cfcb7e4cf10ee
Sep 18 14:05:41.515714 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:41.515681    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx" event={"ID":"e80282ae-09d3-42b0-9cfd-d1495199f99f","Type":"ContainerStarted","Data":"a55bce5b6c59badc2f4329c1c3045bbf633741440642d71689d9daaf100d69bd"}
Sep 18 14:05:41.515714 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:41.515719    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx" event={"ID":"e80282ae-09d3-42b0-9cfd-d1495199f99f","Type":"ContainerStarted","Data":"5abf01f0ebaca2277c488ec2bce187e84f592c6b12b4a38c968cfcb7e4cf10ee"}
Sep 18 14:05:41.533264 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:41.533212    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx" podStartSLOduration=1.533197439 podStartE2EDuration="1.533197439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:05:41.533169673 +0000 UTC m=+1283.435625862" watchObservedRunningTime="2026-09-18 14:05:41.533197439 +0000 UTC m=+1283.435653626"
Sep 18 14:05:50.509560 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:50.509530    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:50.509560 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:50.509567    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:50.522551 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:50.522519    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:05:50.572545 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:05:50.572521    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:06:00.865369 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:00.865337    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"]
Sep 18 14:06:00.865826 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:00.865566    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2" podUID="38ae448b-ec07-4f7b-a997-49dfe36e22b5" containerName="maas-api" containerID="cri-o://28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f" gracePeriod=30
Sep 18 14:06:01.121657 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.121591    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:06:01.219941 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.219911    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7f4x\" (UniqueName: \"kubernetes.io/projected/38ae448b-ec07-4f7b-a997-49dfe36e22b5-kube-api-access-k7f4x\") pod \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") "
Sep 18 14:06:01.220131 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.220007    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-metrics-certs\") pod \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") "
Sep 18 14:06:01.220131 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.220046    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-maas-api-tls\") pod \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") "
Sep 18 14:06:01.220131 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.220101    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ae448b-ec07-4f7b-a997-49dfe36e22b5-openshift-service-ca\") pod \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\" (UID: \"38ae448b-ec07-4f7b-a997-49dfe36e22b5\") "
Sep 18 14:06:01.220467 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.220442    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ae448b-ec07-4f7b-a997-49dfe36e22b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38ae448b-ec07-4f7b-a997-49dfe36e22b5" (UID: "38ae448b-ec07-4f7b-a997-49dfe36e22b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:06:01.222290 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.222267    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ae448b-ec07-4f7b-a997-49dfe36e22b5-kube-api-access-k7f4x" (OuterVolumeSpecName: "kube-api-access-k7f4x") pod "38ae448b-ec07-4f7b-a997-49dfe36e22b5" (UID: "38ae448b-ec07-4f7b-a997-49dfe36e22b5"). InnerVolumeSpecName "kube-api-access-k7f4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:06:01.222385 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.222301    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38ae448b-ec07-4f7b-a997-49dfe36e22b5" (UID: "38ae448b-ec07-4f7b-a997-49dfe36e22b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:06:01.222385 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.222353    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "38ae448b-ec07-4f7b-a997-49dfe36e22b5" (UID: "38ae448b-ec07-4f7b-a997-49dfe36e22b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:06:01.321727 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.321693    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7f4x\" (UniqueName: \"kubernetes.io/projected/38ae448b-ec07-4f7b-a997-49dfe36e22b5-kube-api-access-k7f4x\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:06:01.321727 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.321730    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:06:01.321925 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.321744    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ae448b-ec07-4f7b-a997-49dfe36e22b5-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:06:01.321925 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.321756    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ae448b-ec07-4f7b-a997-49dfe36e22b5-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:06:01.619084 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.619041    2569 generic.go:356] "Generic (PLEG): container finished" podID="38ae448b-ec07-4f7b-a997-49dfe36e22b5" containerID="28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f" exitCode=0
Sep 18 14:06:01.619299 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.619108    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"
Sep 18 14:06:01.619299 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.619125    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2" event={"ID":"38ae448b-ec07-4f7b-a997-49dfe36e22b5","Type":"ContainerDied","Data":"28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f"}
Sep 18 14:06:01.619299 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.619196    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2" event={"ID":"38ae448b-ec07-4f7b-a997-49dfe36e22b5","Type":"ContainerDied","Data":"76fbd626307dee3230abb117d4a4a990b773d4301986ede7f64c26eaaaedcdf7"}
Sep 18 14:06:01.619299 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.619226    2569 scope.go:117] "RemoveContainer" containerID="28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f"
Sep 18 14:06:01.629224 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.629208    2569 scope.go:117] "RemoveContainer" containerID="28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f"
Sep 18 14:06:01.629550 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:06:01.629533    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f\": container with ID starting with 28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f not found: ID does not exist" containerID="28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f"
Sep 18 14:06:01.629609 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.629563    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f"} err="failed to get container status \"28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f\": rpc error: code = NotFound desc = could not find container \"28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f\": container with ID starting with 28ccb920f7588b405668d2ae37de3ec3875b21279e13babe3dc54bf1be30027f not found: ID does not exist"
Sep 18 14:06:01.642335 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.642311    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"]
Sep 18 14:06:01.647436 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:01.647416    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bfd52075-7488dff6bd-rcwg2"]
Sep 18 14:06:02.487227 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:02.487191    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ae448b-ec07-4f7b-a997-49dfe36e22b5" path="/var/lib/kubelet/pods/38ae448b-ec07-4f7b-a997-49dfe36e22b5/volumes"
Sep 18 14:06:18.606895 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:06:18.606866    2569 scope.go:117] "RemoveContainer" containerID="4b0153b15159bb4a35610a9605134dcb3e92e1164075c408a0b28e936873c94f"
Sep 18 14:08:21.486059 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:21.486024    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"]
Sep 18 14:08:21.486593 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:21.486283    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v" podUID="349ba1a7-c10a-43f5-b853-ca6fb6872ba8" containerName="maas-api" containerID="cri-o://2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11" gracePeriod=30
Sep 18 14:08:21.983600 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:21.983563    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 14:08:22.076023 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.075938    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-maas-api-tls\") pod \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") "
Sep 18 14:08:22.076023 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.075998    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-metrics-certs\") pod \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") "
Sep 18 14:08:22.076288 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.076044    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-openshift-service-ca\") pod \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") "
Sep 18 14:08:22.076288 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.076097    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4czbq\" (UniqueName: \"kubernetes.io/projected/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-kube-api-access-4czbq\") pod \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\" (UID: \"349ba1a7-c10a-43f5-b853-ca6fb6872ba8\") "
Sep 18 14:08:22.076443 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.076414    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "349ba1a7-c10a-43f5-b853-ca6fb6872ba8" (UID: "349ba1a7-c10a-43f5-b853-ca6fb6872ba8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:08:22.078280 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.078255    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "349ba1a7-c10a-43f5-b853-ca6fb6872ba8" (UID: "349ba1a7-c10a-43f5-b853-ca6fb6872ba8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:08:22.078280 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.078263    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-kube-api-access-4czbq" (OuterVolumeSpecName: "kube-api-access-4czbq") pod "349ba1a7-c10a-43f5-b853-ca6fb6872ba8" (UID: "349ba1a7-c10a-43f5-b853-ca6fb6872ba8"). InnerVolumeSpecName "kube-api-access-4czbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:08:22.078445 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.078345    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "349ba1a7-c10a-43f5-b853-ca6fb6872ba8" (UID: "349ba1a7-c10a-43f5-b853-ca6fb6872ba8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:08:22.177190 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.177151    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:22.177190 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.177183    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:22.177190 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.177196    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czbq\" (UniqueName: \"kubernetes.io/projected/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-kube-api-access-4czbq\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:22.177421 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.177210    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349ba1a7-c10a-43f5-b853-ca6fb6872ba8-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:22.364899 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.364811    2569 generic.go:356] "Generic (PLEG): container finished" podID="349ba1a7-c10a-43f5-b853-ca6fb6872ba8" containerID="2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11" exitCode=0
Sep 18 14:08:22.364899 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.364857    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"
Sep 18 14:08:22.365155 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.364894    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v" event={"ID":"349ba1a7-c10a-43f5-b853-ca6fb6872ba8","Type":"ContainerDied","Data":"2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11"}
Sep 18 14:08:22.365155 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.364949    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v" event={"ID":"349ba1a7-c10a-43f5-b853-ca6fb6872ba8","Type":"ContainerDied","Data":"5268db1ba4685b4d2c22f35d9a3b81df31e67e6a6f2b8f45d851c7caf4928d47"}
Sep 18 14:08:22.365155 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.364976    2569 scope.go:117] "RemoveContainer" containerID="2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11"
Sep 18 14:08:22.374768 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.374752    2569 scope.go:117] "RemoveContainer" containerID="2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11"
Sep 18 14:08:22.375024 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:08:22.375008    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11\": container with ID starting with 2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11 not found: ID does not exist" containerID="2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11"
Sep 18 14:08:22.375104 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.375032    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11"} err="failed to get container status \"2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11\": rpc error: code = NotFound desc = could not find container \"2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11\": container with ID starting with 2e7e8319ba68f33d3bf041b42eddc790aeeadd0f4971d197bbcd263f6a367a11 not found: ID does not exist"
Sep 18 14:08:22.402107 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.402060    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"]
Sep 18 14:08:22.404681 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.404662    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ed4c53-6544f484b6-9x66v"]
Sep 18 14:08:22.487803 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:22.487777    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="349ba1a7-c10a-43f5-b853-ca6fb6872ba8" path="/var/lib/kubelet/pods/349ba1a7-c10a-43f5-b853-ca6fb6872ba8/volumes"
Sep 18 14:08:30.712465 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:30.712421    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"]
Sep 18 14:08:30.713053 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:30.712683    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy" containerID="cri-o://f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e" gracePeriod=30
Sep 18 14:08:35.971841 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:35.971813    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 14:08:36.098518 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098441    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-token\") pod \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") "
Sep 18 14:08:36.098518 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098484    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-data\") pod \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") "
Sep 18 14:08:36.098518 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098503    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istiod-ca-cert\") pod \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") "
Sep 18 14:08:36.098765 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098537    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-socket\") pod \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") "
Sep 18 14:08:36.098765 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098566    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-podinfo\") pod \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") "
Sep 18 14:08:36.098765 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098635    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-envoy\") pod \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") "
Sep 18 14:08:36.098765 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098695    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcpc4\" (UniqueName: \"kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-kube-api-access-mcpc4\") pod \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") "
Sep 18 14:08:36.098765 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098719    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-credential-socket\") pod \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") "
Sep 18 14:08:36.099096 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098755    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-data" (OuterVolumeSpecName: "istio-data") pod "0241500c-e3f7-43dc-895e-5e3ff4bc0275" (UID: "0241500c-e3f7-43dc-895e-5e3ff4bc0275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:36.099096 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098859    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0241500c-e3f7-43dc-895e-5e3ff4bc0275" (UID: "0241500c-e3f7-43dc-895e-5e3ff4bc0275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:36.099462 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098923    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0241500c-e3f7-43dc-895e-5e3ff4bc0275" (UID: "0241500c-e3f7-43dc-895e-5e3ff4bc0275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:36.099462 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.098970    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0241500c-e3f7-43dc-895e-5e3ff4bc0275" (UID: "0241500c-e3f7-43dc-895e-5e3ff4bc0275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:08:36.099462 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.099055    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-certs\") pod \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\" (UID: \"0241500c-e3f7-43dc-895e-5e3ff4bc0275\") "
Sep 18 14:08:36.099462 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.099227    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0241500c-e3f7-43dc-895e-5e3ff4bc0275" (UID: "0241500c-e3f7-43dc-895e-5e3ff4bc0275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:36.099685 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.099638    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-credential-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:36.099685 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.099661    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:36.099685 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.099676    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-data\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:36.099844 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.099689    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istiod-ca-cert\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:36.099844 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.099712    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-workload-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:36.100991 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.100967    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-token" (OuterVolumeSpecName: "istio-token") pod "0241500c-e3f7-43dc-895e-5e3ff4bc0275" (UID: "0241500c-e3f7-43dc-895e-5e3ff4bc0275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:08:36.101155 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.100989    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0241500c-e3f7-43dc-895e-5e3ff4bc0275" (UID: "0241500c-e3f7-43dc-895e-5e3ff4bc0275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:08:36.101439 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.101419    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0241500c-e3f7-43dc-895e-5e3ff4bc0275" (UID: "0241500c-e3f7-43dc-895e-5e3ff4bc0275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:08:36.101498 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.101456    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-kube-api-access-mcpc4" (OuterVolumeSpecName: "kube-api-access-mcpc4") pod "0241500c-e3f7-43dc-895e-5e3ff4bc0275" (UID: "0241500c-e3f7-43dc-895e-5e3ff4bc0275"). InnerVolumeSpecName "kube-api-access-mcpc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:08:36.200217 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.200193    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-envoy\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:36.200217 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.200217    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcpc4\" (UniqueName: \"kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-kube-api-access-mcpc4\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:36.200344 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.200231    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-token\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:36.200344 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.200242    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0241500c-e3f7-43dc-895e-5e3ff4bc0275-istio-podinfo\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:08:36.443508 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.443422    2569 generic.go:356] "Generic (PLEG): container finished" podID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerID="f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e" exitCode=0
Sep 18 14:08:36.443508 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.443464    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" event={"ID":"0241500c-e3f7-43dc-895e-5e3ff4bc0275","Type":"ContainerDied","Data":"f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e"}
Sep 18 14:08:36.443508 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.443479    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"
Sep 18 14:08:36.443508 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.443506    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4" event={"ID":"0241500c-e3f7-43dc-895e-5e3ff4bc0275","Type":"ContainerDied","Data":"b6060390eeacdff0c07c0dd0762c936b4013768023fddba58b44db36568be45d"}
Sep 18 14:08:36.443826 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.443527    2569 scope.go:117] "RemoveContainer" containerID="f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e"
Sep 18 14:08:36.453147 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.453127    2569 scope.go:117] "RemoveContainer" containerID="3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3"
Sep 18 14:08:36.462364 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.462347    2569 scope.go:117] "RemoveContainer" containerID="f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e"
Sep 18 14:08:36.462624 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:08:36.462607    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e\": container with ID starting with f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e not found: ID does not exist" containerID="f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e"
Sep 18 14:08:36.462669 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.462635    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e"} err="failed to get container status \"f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e\": rpc error: code = NotFound desc = could not find container \"f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e\": container with ID starting with f4778732d783b405b89bcccd9faeb4ece382cff52a8fe9eb13dc7a9f4e17e90e not found: ID does not exist"
Sep 18 14:08:36.462669 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.462651    2569 scope.go:117] "RemoveContainer" containerID="3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3"
Sep 18 14:08:36.462928 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:08:36.462901    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3\": container with ID starting with 3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3 not found: ID does not exist" containerID="3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3"
Sep 18 14:08:36.463025 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.462934    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3"} err="failed to get container status \"3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3\": rpc error: code = NotFound desc = could not find container \"3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3\": container with ID starting with 3f0e58b2e0f3bbf577691673985d95c94127ffc62b3ebf31760e4a99f894fdd3 not found: ID does not exist"
Sep 18 14:08:36.466031 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.466010    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"]
Sep 18 14:08:36.471768 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.471750    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ed4c53-openshift-default-57c6d78c9b-2pbk4"]
Sep 18 14:08:36.485570 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:36.485551    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" path="/var/lib/kubelet/pods/0241500c-e3f7-43dc-895e-5e3ff4bc0275/volumes"
Sep 18 14:08:50.648164 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648129    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"]
Sep 18 14:08:50.648580 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648523    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy"
Sep 18 14:08:50.648580 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648538    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy"
Sep 18 14:08:50.648580 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648554    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="349ba1a7-c10a-43f5-b853-ca6fb6872ba8" containerName="maas-api"
Sep 18 14:08:50.648580 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648560    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="349ba1a7-c10a-43f5-b853-ca6fb6872ba8" containerName="maas-api"
Sep 18 14:08:50.648580 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648580    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ae448b-ec07-4f7b-a997-49dfe36e22b5" containerName="maas-api"
Sep 18 14:08:50.648775 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648585    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ae448b-ec07-4f7b-a997-49dfe36e22b5" containerName="maas-api"
Sep 18 14:08:50.648775 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648649    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="349ba1a7-c10a-43f5-b853-ca6fb6872ba8" containerName="maas-api"
Sep 18 14:08:50.648775 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648658    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy"
Sep 18 14:08:50.648775 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648663    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy"
Sep 18 14:08:50.648775 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.648670    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ae448b-ec07-4f7b-a997-49dfe36e22b5" containerName="maas-api"
Sep 18 14:08:50.651667 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.651635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.654852 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.654834    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-eb3054-openshift-default-dockercfg-p64nz\""
Sep 18 14:08:50.668675 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.668650    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"]
Sep 18 14:08:50.732121 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.732085    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-socket\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.732310 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.732148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-token\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.732310 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.732173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-envoy\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.732310 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.732196    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-certs\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.732310 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.732276    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-data\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.732540 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.732313    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istiod-ca-cert\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.732540 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.732348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5xf2\" (UniqueName: \"kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-kube-api-access-h5xf2\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.732540 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.732376    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-podinfo\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.732540 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.732407    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-credential-socket\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.833665 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.833635    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5xf2\" (UniqueName: \"kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-kube-api-access-h5xf2\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.833837 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.833670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-podinfo\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.833837 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.833699    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-credential-socket\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.833837 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.833755    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-socket\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.833837 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.833783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-token\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.833837 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.833808    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-envoy\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.834244 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.834220    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-credential-socket\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.834351 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.834264    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-socket\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.834351 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.834230    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-certs\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.834476 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.834376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-certs\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.834476 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.834392    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-data\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.834476 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.834426    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istiod-ca-cert\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.834815 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.834672    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-data\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.835405 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.835382    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istiod-ca-cert\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.836465 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.836444    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-envoy\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.836658 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.836631    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-podinfo\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.843662 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.843636    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-token\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.844478 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.844457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5xf2\" (UniqueName: \"kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-kube-api-access-h5xf2\") pod \"e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") " pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:50.963097 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:50.963043    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:51.248275 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:51.248250    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"]
Sep 18 14:08:51.251083 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:08:51.251031    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd4b0fb_8d52_4a32_a447_9b65c7d43cf5.slice/crio-3543278708d78791461df5ab771638baf5b7bea8d23a7db70834a896297dbf73 WatchSource:0}: Error finding container 3543278708d78791461df5ab771638baf5b7bea8d23a7db70834a896297dbf73: Status 404 returned error can't find the container with id 3543278708d78791461df5ab771638baf5b7bea8d23a7db70834a896297dbf73
Sep 18 14:08:51.253473 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:51.253432    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:08:51.253578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:51.253520    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:08:51.253578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:51.253562    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:08:51.522833 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:51.522745    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" event={"ID":"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5","Type":"ContainerStarted","Data":"5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e"}
Sep 18 14:08:51.522833 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:51.522795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" event={"ID":"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5","Type":"ContainerStarted","Data":"3543278708d78791461df5ab771638baf5b7bea8d23a7db70834a896297dbf73"}
Sep 18 14:08:51.545769 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:51.545720    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" podStartSLOduration=1.545703687 podStartE2EDuration="1.545703687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:08:51.544216033 +0000 UTC m=+1473.446672219" watchObservedRunningTime="2026-09-18 14:08:51.545703687 +0000 UTC m=+1473.448159873"
Sep 18 14:08:51.963587 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:51.963556    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:52.969872 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:52.969829    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:53.533715 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:53.533685    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:53.534721 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:53.534702    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:08:55.306010 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.305919    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"]
Sep 18 14:08:55.306455 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.306376    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy"
Sep 18 14:08:55.306455 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.306390    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0241500c-e3f7-43dc-895e-5e3ff4bc0275" containerName="istio-proxy"
Sep 18 14:08:55.309776 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.309758    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.312869 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.312850    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-eb3054-serving-cert\""
Sep 18 14:08:55.323426 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.323396    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"]
Sep 18 14:08:55.384361 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.384318    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.384361 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.384358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-metrics-certs\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.384609 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.384491    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55445413-b746-48fd-b78b-f58aaee8a1e1-openshift-service-ca\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.384609 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.384510    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2x7q\" (UniqueName: \"kubernetes.io/projected/55445413-b746-48fd-b78b-f58aaee8a1e1-kube-api-access-h2x7q\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.485700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.485663    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.485890 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.485721    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-metrics-certs\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.485890 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:08:55.485830    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-serving-cert: secret "maas-api-e2e-api-b-eb3054-serving-cert" not found
Sep 18 14:08:55.486003 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:08:55.485910    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls podName:55445413-b746-48fd-b78b-f58aaee8a1e1 nodeName:}" failed. No retries permitted until 2026-09-18 14:08:55.985888183 +0000 UTC m=+1477.888344371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls") pod "maas-api-e2e-api-b-eb3054-889c849b8-4q64h" (UID: "55445413-b746-48fd-b78b-f58aaee8a1e1") : secret "maas-api-e2e-api-b-eb3054-serving-cert" not found
Sep 18 14:08:55.486003 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.485836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55445413-b746-48fd-b78b-f58aaee8a1e1-openshift-service-ca\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.486003 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.485994    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2x7q\" (UniqueName: \"kubernetes.io/projected/55445413-b746-48fd-b78b-f58aaee8a1e1-kube-api-access-h2x7q\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.486560 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.486538    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55445413-b746-48fd-b78b-f58aaee8a1e1-openshift-service-ca\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.488559 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.488542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-metrics-certs\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.499799 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.495140    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2x7q\" (UniqueName: \"kubernetes.io/projected/55445413-b746-48fd-b78b-f58aaee8a1e1-kube-api-access-h2x7q\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.992334 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:55.992203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:55.992531 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:08:55.992391    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-serving-cert: secret "maas-api-e2e-api-b-eb3054-serving-cert" not found
Sep 18 14:08:55.992531 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:08:55.992470    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls podName:55445413-b746-48fd-b78b-f58aaee8a1e1 nodeName:}" failed. No retries permitted until 2026-09-18 14:08:56.992449348 +0000 UTC m=+1478.894905518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls") pod "maas-api-e2e-api-b-eb3054-889c849b8-4q64h" (UID: "55445413-b746-48fd-b78b-f58aaee8a1e1") : secret "maas-api-e2e-api-b-eb3054-serving-cert" not found
Sep 18 14:08:57.002898 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:57.002864    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:57.005505 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:57.005482    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls\") pod \"maas-api-e2e-api-b-eb3054-889c849b8-4q64h\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:57.123929 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:57.123897    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:57.397018 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:57.396995    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"]
Sep 18 14:08:57.399224 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:08:57.399197    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55445413_b746_48fd_b78b_f58aaee8a1e1.slice/crio-e86f608e3912ad2387c4712def3579296297dec50619987f3b89420d719203ff WatchSource:0}: Error finding container e86f608e3912ad2387c4712def3579296297dec50619987f3b89420d719203ff: Status 404 returned error can't find the container with id e86f608e3912ad2387c4712def3579296297dec50619987f3b89420d719203ff
Sep 18 14:08:57.400516 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:57.400498    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:08:57.556372 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:57.556327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h" event={"ID":"55445413-b746-48fd-b78b-f58aaee8a1e1","Type":"ContainerStarted","Data":"e86f608e3912ad2387c4712def3579296297dec50619987f3b89420d719203ff"}
Sep 18 14:08:58.562600 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:58.562566    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h" event={"ID":"55445413-b746-48fd-b78b-f58aaee8a1e1","Type":"ContainerStarted","Data":"5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185"}
Sep 18 14:08:58.562980 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:58.562628    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:08:58.586019 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:08:58.585975    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h" podStartSLOduration=3.262136142 podStartE2EDuration="3.585959513s" totalImagesPullingTime="323.823371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:08:58.584997093 +0000 UTC m=+1480.487453281" watchObservedRunningTime="2026-09-18 14:08:58.585959513 +0000 UTC m=+1480.488415696"
Sep 18 14:09:04.573322 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:04.573227    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:09:14.647294 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:14.647212    2569 generic.go:356] "Generic (PLEG): container finished" podID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerID="5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e" exitCode=0
Sep 18 14:09:14.647674 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:14.647289    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" event={"ID":"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5","Type":"ContainerDied","Data":"5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e"}
Sep 18 14:09:14.647766 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:14.647747    2569 scope.go:117] "RemoveContainer" containerID="5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e"
Sep 18 14:09:14.648899 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:14.648873    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:09:14.648953 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:14.648940    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:09:14.648992 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:14.648975    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:09:15.657777 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:15.657734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" event={"ID":"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5","Type":"ContainerStarted","Data":"6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1"}
Sep 18 14:09:15.963262 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:15.963227    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:09:15.963502 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:15.963476    2569 patch_prober.go:28] interesting pod/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 14:09:15.963588 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:15.963537    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 14:09:16.963540 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:16.963498    2569 patch_prober.go:28] interesting pod/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 14:09:16.964615 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:16.964185    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 14:09:17.963836 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:17.963798    2569 patch_prober.go:28] interesting pod/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 14:09:17.964329 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:17.963876    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 14:09:18.585023 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:18.584994    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 14:09:18.585563 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:18.585542    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 14:09:18.964063 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:18.964028    2569 patch_prober.go:28] interesting pod/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 14:09:18.964508 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:18.964119    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 14:09:19.964352 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:19.964312    2569 patch_prober.go:28] interesting pod/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 14:09:19.964764 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:19.964403    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 14:09:20.989055 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:20.989022    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:09:20.990199 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:20.990121    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:09:20.991357 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:20.991338    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:09:48.692031 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:48.691950    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"]
Sep 18 14:09:55.385450 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.385359    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"]
Sep 18 14:09:55.385883 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.385682    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g" podUID="ed1c0dd6-c519-4bc5-8b09-c7d30399073d" containerName="maas-api" containerID="cri-o://e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99" gracePeriod=30
Sep 18 14:09:55.446299 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.446261    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"]
Sep 18 14:09:55.446550 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.446502    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg" podUID="2848a5b6-e7ed-4fbc-a750-556cfea2a76a" containerName="payload-processing" containerID="cri-o://708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845" gracePeriod=30
Sep 18 14:09:55.512992 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.512940    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g" podUID="ed1c0dd6-c519-4bc5-8b09-c7d30399073d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.34:8443/health\": dial tcp 10.133.0.34:8443: connect: connection refused"
Sep 18 14:09:55.718757 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.718731    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 14:09:55.750300 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.750276    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 14:09:55.800483 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.800451    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz7xm\" (UniqueName: \"kubernetes.io/projected/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-kube-api-access-tz7xm\") pod \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") "
Sep 18 14:09:55.800655 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.800507    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-plugins-config-volume\") pod \"2848a5b6-e7ed-4fbc-a750-556cfea2a76a\" (UID: \"2848a5b6-e7ed-4fbc-a750-556cfea2a76a\") "
Sep 18 14:09:55.800655 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.800535    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gntmg\" (UniqueName: \"kubernetes.io/projected/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-kube-api-access-gntmg\") pod \"2848a5b6-e7ed-4fbc-a750-556cfea2a76a\" (UID: \"2848a5b6-e7ed-4fbc-a750-556cfea2a76a\") "
Sep 18 14:09:55.800655 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.800553    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-metrics-certs\") pod \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") "
Sep 18 14:09:55.800655 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.800589    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-maas-api-tls\") pod \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") "
Sep 18 14:09:55.800883 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.800773    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-openshift-service-ca\") pod \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\" (UID: \"ed1c0dd6-c519-4bc5-8b09-c7d30399073d\") "
Sep 18 14:09:55.801150 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.801120    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed1c0dd6-c519-4bc5-8b09-c7d30399073d" (UID: "ed1c0dd6-c519-4bc5-8b09-c7d30399073d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:09:55.801448 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.801228    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:09:55.801448 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.801296    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2848a5b6-e7ed-4fbc-a750-556cfea2a76a" (UID: "2848a5b6-e7ed-4fbc-a750-556cfea2a76a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:09:55.802705 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.802685    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-kube-api-access-tz7xm" (OuterVolumeSpecName: "kube-api-access-tz7xm") pod "ed1c0dd6-c519-4bc5-8b09-c7d30399073d" (UID: "ed1c0dd6-c519-4bc5-8b09-c7d30399073d"). InnerVolumeSpecName "kube-api-access-tz7xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:09:55.802869 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.802851    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ed1c0dd6-c519-4bc5-8b09-c7d30399073d" (UID: "ed1c0dd6-c519-4bc5-8b09-c7d30399073d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:09:55.803329 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.803303    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed1c0dd6-c519-4bc5-8b09-c7d30399073d" (UID: "ed1c0dd6-c519-4bc5-8b09-c7d30399073d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:09:55.803329 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.803315    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-kube-api-access-gntmg" (OuterVolumeSpecName: "kube-api-access-gntmg") pod "2848a5b6-e7ed-4fbc-a750-556cfea2a76a" (UID: "2848a5b6-e7ed-4fbc-a750-556cfea2a76a"). InnerVolumeSpecName "kube-api-access-gntmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:09:55.880652 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.880621    2569 generic.go:356] "Generic (PLEG): container finished" podID="ed1c0dd6-c519-4bc5-8b09-c7d30399073d" containerID="e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99" exitCode=0
Sep 18 14:09:55.880827 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.880667    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"
Sep 18 14:09:55.880827 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.880709    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g" event={"ID":"ed1c0dd6-c519-4bc5-8b09-c7d30399073d","Type":"ContainerDied","Data":"e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99"}
Sep 18 14:09:55.880827 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.880764    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g" event={"ID":"ed1c0dd6-c519-4bc5-8b09-c7d30399073d","Type":"ContainerDied","Data":"db07b56daa272abe0aea9edc5921f0795dcf5d3fe3f9dc3386373a0a17df2e7f"}
Sep 18 14:09:55.880827 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.880800    2569 scope.go:117] "RemoveContainer" containerID="e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99"
Sep 18 14:09:55.882177 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.882153    2569 generic.go:356] "Generic (PLEG): container finished" podID="2848a5b6-e7ed-4fbc-a750-556cfea2a76a" containerID="708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845" exitCode=0
Sep 18 14:09:55.882304 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.882184    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg" event={"ID":"2848a5b6-e7ed-4fbc-a750-556cfea2a76a","Type":"ContainerDied","Data":"708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845"}
Sep 18 14:09:55.882304 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.882243    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg" event={"ID":"2848a5b6-e7ed-4fbc-a750-556cfea2a76a","Type":"ContainerDied","Data":"0015622219fceeb1d2211c73407b82662e8094509d041ac199a65414db9b462b"}
Sep 18 14:09:55.882304 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.882193    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"
Sep 18 14:09:55.891159 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.891137    2569 scope.go:117] "RemoveContainer" containerID="e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99"
Sep 18 14:09:55.891498 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:09:55.891478    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99\": container with ID starting with e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99 not found: ID does not exist" containerID="e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99"
Sep 18 14:09:55.891581 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.891503    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99"} err="failed to get container status \"e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99\": rpc error: code = NotFound desc = could not find container \"e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99\": container with ID starting with e4d881178efb9b4caa6784e95086636e426ac2f645783081e3bb40ddda639f99 not found: ID does not exist"
Sep 18 14:09:55.891581 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.891520    2569 scope.go:117] "RemoveContainer" containerID="708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845"
Sep 18 14:09:55.899582 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.899563    2569 scope.go:117] "RemoveContainer" containerID="708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845"
Sep 18 14:09:55.899827 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:09:55.899808    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845\": container with ID starting with 708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845 not found: ID does not exist" containerID="708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845"
Sep 18 14:09:55.899879 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.899834    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845"} err="failed to get container status \"708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845\": rpc error: code = NotFound desc = could not find container \"708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845\": container with ID starting with 708004eb9f7f5789bc809953db7b3bf5d9327a6354ca41b73a2636087d7eb845 not found: ID does not exist"
Sep 18 14:09:55.902514 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.902497    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:09:55.902514 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.902514    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gntmg\" (UniqueName: \"kubernetes.io/projected/2848a5b6-e7ed-4fbc-a750-556cfea2a76a-kube-api-access-gntmg\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:09:55.902713 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.902531    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:09:55.902713 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.902541    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:09:55.902713 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.902548    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz7xm\" (UniqueName: \"kubernetes.io/projected/ed1c0dd6-c519-4bc5-8b09-c7d30399073d-kube-api-access-tz7xm\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:09:55.905384 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.905363    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"]
Sep 18 14:09:55.909733 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.909674    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-377ca2-78868c6cff-ggq8g"]
Sep 18 14:09:55.922130 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.922108    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"]
Sep 18 14:09:55.927780 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:55.927762    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-377ca2-ccf555f7c-hr8cg"]
Sep 18 14:09:56.487904 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:56.487871    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2848a5b6-e7ed-4fbc-a750-556cfea2a76a" path="/var/lib/kubelet/pods/2848a5b6-e7ed-4fbc-a750-556cfea2a76a/volumes"
Sep 18 14:09:56.488338 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:09:56.488323    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed1c0dd6-c519-4bc5-8b09-c7d30399073d" path="/var/lib/kubelet/pods/ed1c0dd6-c519-4bc5-8b09-c7d30399073d/volumes"
Sep 18 14:10:03.711391 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:03.711354    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx" podUID="e80282ae-09d3-42b0-9cfd-d1495199f99f" containerName="main" containerID="cri-o://a55bce5b6c59badc2f4329c1c3045bbf633741440642d71689d9daaf100d69bd" gracePeriod=45
Sep 18 14:10:03.930003 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:03.929974    2569 generic.go:356] "Generic (PLEG): container finished" podID="e80282ae-09d3-42b0-9cfd-d1495199f99f" containerID="a55bce5b6c59badc2f4329c1c3045bbf633741440642d71689d9daaf100d69bd" exitCode=0
Sep 18 14:10:03.930183 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:03.930049    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx" event={"ID":"e80282ae-09d3-42b0-9cfd-d1495199f99f","Type":"ContainerDied","Data":"a55bce5b6c59badc2f4329c1c3045bbf633741440642d71689d9daaf100d69bd"}
Sep 18 14:10:03.968541 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:03.968485    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:10:04.082588 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.082558    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-home\") pod \"e80282ae-09d3-42b0-9cfd-d1495199f99f\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") "
Sep 18 14:10:04.082779 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.082610    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80282ae-09d3-42b0-9cfd-d1495199f99f-tls-certs\") pod \"e80282ae-09d3-42b0-9cfd-d1495199f99f\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") "
Sep 18 14:10:04.082779 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.082680    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j49t\" (UniqueName: \"kubernetes.io/projected/e80282ae-09d3-42b0-9cfd-d1495199f99f-kube-api-access-5j49t\") pod \"e80282ae-09d3-42b0-9cfd-d1495199f99f\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") "
Sep 18 14:10:04.082779 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.082708    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-dshm\") pod \"e80282ae-09d3-42b0-9cfd-d1495199f99f\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") "
Sep 18 14:10:04.082959 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.082787    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-tmp-dir\") pod \"e80282ae-09d3-42b0-9cfd-d1495199f99f\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") "
Sep 18 14:10:04.082959 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.082810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-model-cache\") pod \"e80282ae-09d3-42b0-9cfd-d1495199f99f\" (UID: \"e80282ae-09d3-42b0-9cfd-d1495199f99f\") "
Sep 18 14:10:04.082959 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.082847    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-home" (OuterVolumeSpecName: "home") pod "e80282ae-09d3-42b0-9cfd-d1495199f99f" (UID: "e80282ae-09d3-42b0-9cfd-d1495199f99f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:04.083174 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.082999    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e80282ae-09d3-42b0-9cfd-d1495199f99f" (UID: "e80282ae-09d3-42b0-9cfd-d1495199f99f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:04.083174 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.083110    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e80282ae-09d3-42b0-9cfd-d1495199f99f" (UID: "e80282ae-09d3-42b0-9cfd-d1495199f99f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:04.083279 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.083255    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-home\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:04.083279 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.083274    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-tmp-dir\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:04.083346 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.083288    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-model-cache\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:04.085211 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.085184    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-dshm" (OuterVolumeSpecName: "dshm") pod "e80282ae-09d3-42b0-9cfd-d1495199f99f" (UID: "e80282ae-09d3-42b0-9cfd-d1495199f99f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:04.085335 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.085232    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80282ae-09d3-42b0-9cfd-d1495199f99f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e80282ae-09d3-42b0-9cfd-d1495199f99f" (UID: "e80282ae-09d3-42b0-9cfd-d1495199f99f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:10:04.085335 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.085262    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80282ae-09d3-42b0-9cfd-d1495199f99f-kube-api-access-5j49t" (OuterVolumeSpecName: "kube-api-access-5j49t") pod "e80282ae-09d3-42b0-9cfd-d1495199f99f" (UID: "e80282ae-09d3-42b0-9cfd-d1495199f99f"). InnerVolumeSpecName "kube-api-access-5j49t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:04.184116 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.184087    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80282ae-09d3-42b0-9cfd-d1495199f99f-dshm\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:04.184116 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.184113    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80282ae-09d3-42b0-9cfd-d1495199f99f-tls-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:04.184308 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.184125    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j49t\" (UniqueName: \"kubernetes.io/projected/e80282ae-09d3-42b0-9cfd-d1495199f99f-kube-api-access-5j49t\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:04.936354 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.936265    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"
Sep 18 14:10:04.936354 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.936294    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx" event={"ID":"e80282ae-09d3-42b0-9cfd-d1495199f99f","Type":"ContainerDied","Data":"5abf01f0ebaca2277c488ec2bce187e84f592c6b12b4a38c968cfcb7e4cf10ee"}
Sep 18 14:10:04.936354 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.936341    2569 scope.go:117] "RemoveContainer" containerID="a55bce5b6c59badc2f4329c1c3045bbf633741440642d71689d9daaf100d69bd"
Sep 18 14:10:04.954452 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.954427    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"]
Sep 18 14:10:04.958187 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:04.958164    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1a4b6d/auth-test-model-1a4b6d-kserve-965df48f9-7dbsx"]
Sep 18 14:10:05.471003 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.470965    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"]
Sep 18 14:10:05.471279 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.471251    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h" podUID="55445413-b746-48fd-b78b-f58aaee8a1e1" containerName="maas-api" containerID="cri-o://5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185" gracePeriod=30
Sep 18 14:10:05.725181 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.725131    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:10:05.798403 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.798370    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-metrics-certs\") pod \"55445413-b746-48fd-b78b-f58aaee8a1e1\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") "
Sep 18 14:10:05.798596 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.798417    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55445413-b746-48fd-b78b-f58aaee8a1e1-openshift-service-ca\") pod \"55445413-b746-48fd-b78b-f58aaee8a1e1\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") "
Sep 18 14:10:05.798596 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.798506    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2x7q\" (UniqueName: \"kubernetes.io/projected/55445413-b746-48fd-b78b-f58aaee8a1e1-kube-api-access-h2x7q\") pod \"55445413-b746-48fd-b78b-f58aaee8a1e1\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") "
Sep 18 14:10:05.798596 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.798583    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls\") pod \"55445413-b746-48fd-b78b-f58aaee8a1e1\" (UID: \"55445413-b746-48fd-b78b-f58aaee8a1e1\") "
Sep 18 14:10:05.798842 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.798797    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55445413-b746-48fd-b78b-f58aaee8a1e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55445413-b746-48fd-b78b-f58aaee8a1e1" (UID: "55445413-b746-48fd-b78b-f58aaee8a1e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:10:05.798954 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.798928    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55445413-b746-48fd-b78b-f58aaee8a1e1-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:05.801411 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.801383    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55445413-b746-48fd-b78b-f58aaee8a1e1-kube-api-access-h2x7q" (OuterVolumeSpecName: "kube-api-access-h2x7q") pod "55445413-b746-48fd-b78b-f58aaee8a1e1" (UID: "55445413-b746-48fd-b78b-f58aaee8a1e1"). InnerVolumeSpecName "kube-api-access-h2x7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:05.801505 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.801415    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55445413-b746-48fd-b78b-f58aaee8a1e1" (UID: "55445413-b746-48fd-b78b-f58aaee8a1e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:10:05.801505 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.801431    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "55445413-b746-48fd-b78b-f58aaee8a1e1" (UID: "55445413-b746-48fd-b78b-f58aaee8a1e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:10:05.899565 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.899530    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:05.899565 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.899558    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55445413-b746-48fd-b78b-f58aaee8a1e1-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:05.899565 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.899568    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2x7q\" (UniqueName: \"kubernetes.io/projected/55445413-b746-48fd-b78b-f58aaee8a1e1-kube-api-access-h2x7q\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:05.943771 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.943734    2569 generic.go:356] "Generic (PLEG): container finished" podID="55445413-b746-48fd-b78b-f58aaee8a1e1" containerID="5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185" exitCode=0
Sep 18 14:10:05.944254 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.943795    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"
Sep 18 14:10:05.944254 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.943823    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h" event={"ID":"55445413-b746-48fd-b78b-f58aaee8a1e1","Type":"ContainerDied","Data":"5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185"}
Sep 18 14:10:05.944254 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.943886    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h" event={"ID":"55445413-b746-48fd-b78b-f58aaee8a1e1","Type":"ContainerDied","Data":"e86f608e3912ad2387c4712def3579296297dec50619987f3b89420d719203ff"}
Sep 18 14:10:05.944254 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.943923    2569 scope.go:117] "RemoveContainer" containerID="5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185"
Sep 18 14:10:05.958620 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.958602    2569 scope.go:117] "RemoveContainer" containerID="5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185"
Sep 18 14:10:05.958905 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:10:05.958889    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185\": container with ID starting with 5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185 not found: ID does not exist" containerID="5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185"
Sep 18 14:10:05.958983 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.958913    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185"} err="failed to get container status \"5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185\": rpc error: code = NotFound desc = could not find container \"5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185\": container with ID starting with 5a7067f3d186e204cca3e3c51cd888a2d329b0a13d3e8275c05719d9c0016185 not found: ID does not exist"
Sep 18 14:10:05.973614 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.973590    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"]
Sep 18 14:10:05.976103 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:05.976042    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb3054-889c849b8-4q64h"]
Sep 18 14:10:06.486625 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:06.486589    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55445413-b746-48fd-b78b-f58aaee8a1e1" path="/var/lib/kubelet/pods/55445413-b746-48fd-b78b-f58aaee8a1e1/volumes"
Sep 18 14:10:06.487108 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:06.487088    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80282ae-09d3-42b0-9cfd-d1495199f99f" path="/var/lib/kubelet/pods/e80282ae-09d3-42b0-9cfd-d1495199f99f/volumes"
Sep 18 14:10:12.488338 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:12.488308    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"]
Sep 18 14:10:12.488778 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:12.488544    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy" containerID="cri-o://6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1" gracePeriod=30
Sep 18 14:10:17.733211 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.733180    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:10:17.816148 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816044    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-envoy\") pod \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") "
Sep 18 14:10:17.816148 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816116    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-data\") pod \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") "
Sep 18 14:10:17.816366 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816159    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-certs\") pod \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") "
Sep 18 14:10:17.816366 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816199    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5xf2\" (UniqueName: \"kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-kube-api-access-h5xf2\") pod \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") "
Sep 18 14:10:17.816366 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816262    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-podinfo\") pod \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") "
Sep 18 14:10:17.816366 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816289    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istiod-ca-cert\") pod \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") "
Sep 18 14:10:17.816366 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816339    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-socket\") pod \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") "
Sep 18 14:10:17.816366 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816364    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-credential-socket\") pod \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") "
Sep 18 14:10:17.816659 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816373    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-data" (OuterVolumeSpecName: "istio-data") pod "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" (UID: "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:17.816659 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816474    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-token\") pod \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\" (UID: \"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5\") "
Sep 18 14:10:17.816659 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816500    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" (UID: "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:17.816659 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816605    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" (UID: "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:17.816843 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816734    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" (UID: "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:17.816843 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816741    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" (UID: "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:10:17.817053 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.816949    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-data\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:17.817161 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.817061    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:17.817161 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.817102    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istiod-ca-cert\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:17.817161 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.817115    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-workload-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:17.817161 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.817128    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-credential-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:17.818571 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.818545    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-kube-api-access-h5xf2" (OuterVolumeSpecName: "kube-api-access-h5xf2") pod "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" (UID: "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5"). InnerVolumeSpecName "kube-api-access-h5xf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:17.818696 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.818607    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-token" (OuterVolumeSpecName: "istio-token") pod "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" (UID: "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:17.818928 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.818909    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" (UID: "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:17.818928 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.818915    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" (UID: "9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:10:17.918587 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.918553    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-token\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:17.918587 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.918582    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-envoy\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:17.918587 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.918595    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5xf2\" (UniqueName: \"kubernetes.io/projected/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-kube-api-access-h5xf2\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:17.918861 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:17.918605    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5-istio-podinfo\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:18.008463 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.008424    2569 generic.go:356] "Generic (PLEG): container finished" podID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerID="6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1" exitCode=0
Sep 18 14:10:18.008641 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.008478    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"
Sep 18 14:10:18.008641 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.008514    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" event={"ID":"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5","Type":"ContainerDied","Data":"6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1"}
Sep 18 14:10:18.008641 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.008572    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c" event={"ID":"9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5","Type":"ContainerDied","Data":"3543278708d78791461df5ab771638baf5b7bea8d23a7db70834a896297dbf73"}
Sep 18 14:10:18.008641 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.008595    2569 scope.go:117] "RemoveContainer" containerID="6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1"
Sep 18 14:10:18.017831 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.017814    2569 scope.go:117] "RemoveContainer" containerID="5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e"
Sep 18 14:10:18.025567 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.025551    2569 scope.go:117] "RemoveContainer" containerID="6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1"
Sep 18 14:10:18.025798 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:10:18.025781    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1\": container with ID starting with 6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1 not found: ID does not exist" containerID="6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1"
Sep 18 14:10:18.025845 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.025807    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1"} err="failed to get container status \"6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1\": rpc error: code = NotFound desc = could not find container \"6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1\": container with ID starting with 6901c7fe5e23df9b60ca6d703d85e92893151badf07902648bd9fb0e0274e3d1 not found: ID does not exist"
Sep 18 14:10:18.025845 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.025823    2569 scope.go:117] "RemoveContainer" containerID="5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e"
Sep 18 14:10:18.026142 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:10:18.026125    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e\": container with ID starting with 5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e not found: ID does not exist" containerID="5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e"
Sep 18 14:10:18.026218 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.026145    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e"} err="failed to get container status \"5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e\": rpc error: code = NotFound desc = could not find container \"5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e\": container with ID starting with 5c34216b77cfecf5231fb60ac34cd87701b927d7c59be2f3dbad6798331ff04e not found: ID does not exist"
Sep 18 14:10:18.030971 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.030950    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"]
Sep 18 14:10:18.037711 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.037692    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-eb3054-openshift-default-58876c59cb-qxb8c"]
Sep 18 14:10:18.486420 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:18.486385    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" path="/var/lib/kubelet/pods/9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5/volumes"
Sep 18 14:10:23.378528 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.377818    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"]
Sep 18 14:10:23.378528 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.378109    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2" podUID="998e310e-0a23-49a3-bc01-f5c827b20c08" containerName="maas-api" containerID="cri-o://dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab" gracePeriod=30
Sep 18 14:10:23.438047 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.438010    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"]
Sep 18 14:10:23.439103 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.438302    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq" podUID="7c48bddc-6e45-430f-97be-753cf5efd021" containerName="payload-pre-processing" containerID="cri-o://317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114" gracePeriod=30
Sep 18 14:10:23.714188 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.714153    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:10:23.736202 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.736174    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:10:23.875333 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.875304    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-metrics-certs\") pod \"998e310e-0a23-49a3-bc01-f5c827b20c08\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") "
Sep 18 14:10:23.875594 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.875374    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zngfl\" (UniqueName: \"kubernetes.io/projected/998e310e-0a23-49a3-bc01-f5c827b20c08-kube-api-access-zngfl\") pod \"998e310e-0a23-49a3-bc01-f5c827b20c08\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") "
Sep 18 14:10:23.875594 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.875407    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dptbv\" (UniqueName: \"kubernetes.io/projected/7c48bddc-6e45-430f-97be-753cf5efd021-kube-api-access-dptbv\") pod \"7c48bddc-6e45-430f-97be-753cf5efd021\" (UID: \"7c48bddc-6e45-430f-97be-753cf5efd021\") "
Sep 18 14:10:23.875594 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.875423    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-maas-api-tls\") pod \"998e310e-0a23-49a3-bc01-f5c827b20c08\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") "
Sep 18 14:10:23.875594 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.875455    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/998e310e-0a23-49a3-bc01-f5c827b20c08-openshift-service-ca\") pod \"998e310e-0a23-49a3-bc01-f5c827b20c08\" (UID: \"998e310e-0a23-49a3-bc01-f5c827b20c08\") "
Sep 18 14:10:23.875594 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.875479    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c48bddc-6e45-430f-97be-753cf5efd021-plugins-config-volume\") pod \"7c48bddc-6e45-430f-97be-753cf5efd021\" (UID: \"7c48bddc-6e45-430f-97be-753cf5efd021\") "
Sep 18 14:10:23.875885 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.875801    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998e310e-0a23-49a3-bc01-f5c827b20c08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "998e310e-0a23-49a3-bc01-f5c827b20c08" (UID: "998e310e-0a23-49a3-bc01-f5c827b20c08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:10:23.875943 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.875894    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c48bddc-6e45-430f-97be-753cf5efd021-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c48bddc-6e45-430f-97be-753cf5efd021" (UID: "7c48bddc-6e45-430f-97be-753cf5efd021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:10:23.877864 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.877837    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "998e310e-0a23-49a3-bc01-f5c827b20c08" (UID: "998e310e-0a23-49a3-bc01-f5c827b20c08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:10:23.877864 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.877838    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998e310e-0a23-49a3-bc01-f5c827b20c08-kube-api-access-zngfl" (OuterVolumeSpecName: "kube-api-access-zngfl") pod "998e310e-0a23-49a3-bc01-f5c827b20c08" (UID: "998e310e-0a23-49a3-bc01-f5c827b20c08"). InnerVolumeSpecName "kube-api-access-zngfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:23.878159 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.878142    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48bddc-6e45-430f-97be-753cf5efd021-kube-api-access-dptbv" (OuterVolumeSpecName: "kube-api-access-dptbv") pod "7c48bddc-6e45-430f-97be-753cf5efd021" (UID: "7c48bddc-6e45-430f-97be-753cf5efd021"). InnerVolumeSpecName "kube-api-access-dptbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:23.878159 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.878141    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "998e310e-0a23-49a3-bc01-f5c827b20c08" (UID: "998e310e-0a23-49a3-bc01-f5c827b20c08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:10:23.976840 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.976810    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/998e310e-0a23-49a3-bc01-f5c827b20c08-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:23.976840 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.976836    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c48bddc-6e45-430f-97be-753cf5efd021-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:23.976840 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.976846    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:23.977052 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.976854    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngfl\" (UniqueName: \"kubernetes.io/projected/998e310e-0a23-49a3-bc01-f5c827b20c08-kube-api-access-zngfl\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:23.977052 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.976862    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dptbv\" (UniqueName: \"kubernetes.io/projected/7c48bddc-6e45-430f-97be-753cf5efd021-kube-api-access-dptbv\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:23.977052 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:23.976869    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/998e310e-0a23-49a3-bc01-f5c827b20c08-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:24.045754 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.045727    2569 generic.go:356] "Generic (PLEG): container finished" podID="7c48bddc-6e45-430f-97be-753cf5efd021" containerID="317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114" exitCode=0
Sep 18 14:10:24.045927 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.045777    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"
Sep 18 14:10:24.045927 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.045799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq" event={"ID":"7c48bddc-6e45-430f-97be-753cf5efd021","Type":"ContainerDied","Data":"317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114"}
Sep 18 14:10:24.045927 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.045849    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq" event={"ID":"7c48bddc-6e45-430f-97be-753cf5efd021","Type":"ContainerDied","Data":"914d4ed40a7ec0b52e549a5792f4a0e6b5ea69f870db51a84abb1bb7baaa6c21"}
Sep 18 14:10:24.045927 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.045880    2569 scope.go:117] "RemoveContainer" containerID="317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114"
Sep 18 14:10:24.047121 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.047100    2569 generic.go:356] "Generic (PLEG): container finished" podID="998e310e-0a23-49a3-bc01-f5c827b20c08" containerID="dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab" exitCode=0
Sep 18 14:10:24.047253 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.047146    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"
Sep 18 14:10:24.047253 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.047238    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2" event={"ID":"998e310e-0a23-49a3-bc01-f5c827b20c08","Type":"ContainerDied","Data":"dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab"}
Sep 18 14:10:24.047376 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.047290    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2" event={"ID":"998e310e-0a23-49a3-bc01-f5c827b20c08","Type":"ContainerDied","Data":"e5d663195762b68683bfc9c4ac77ef751bac14b8bed0fad413967d1d651dafc3"}
Sep 18 14:10:24.056240 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.056175    2569 scope.go:117] "RemoveContainer" containerID="317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114"
Sep 18 14:10:24.056513 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:10:24.056491    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114\": container with ID starting with 317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114 not found: ID does not exist" containerID="317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114"
Sep 18 14:10:24.056581 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.056520    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114"} err="failed to get container status \"317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114\": rpc error: code = NotFound desc = could not find container \"317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114\": container with ID starting with 317b7901c33ba5e55ad3bec862f07134d0b8c2cf9f8aebb9d7833464d09a9114 not found: ID does not exist"
Sep 18 14:10:24.056581 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.056538    2569 scope.go:117] "RemoveContainer" containerID="dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab"
Sep 18 14:10:24.064886 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.064871    2569 scope.go:117] "RemoveContainer" containerID="dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab"
Sep 18 14:10:24.065220 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:10:24.065202    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab\": container with ID starting with dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab not found: ID does not exist" containerID="dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab"
Sep 18 14:10:24.065268 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.065226    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab"} err="failed to get container status \"dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab\": rpc error: code = NotFound desc = could not find container \"dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab\": container with ID starting with dd4fed6eba37070b6ef48148697e39450119f150329a7047d9cdb6274f68d3ab not found: ID does not exist"
Sep 18 14:10:24.069006 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.068987    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"]
Sep 18 14:10:24.073540 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.073520    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e46fbf-6d7f57c9fd-f2khq"]
Sep 18 14:10:24.086783 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.086761    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"]
Sep 18 14:10:24.091477 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.091456    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e46fbf-54f6ccd58d-bwwj2"]
Sep 18 14:10:24.486776 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.486746    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c48bddc-6e45-430f-97be-753cf5efd021" path="/var/lib/kubelet/pods/7c48bddc-6e45-430f-97be-753cf5efd021/volumes"
Sep 18 14:10:24.487155 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:24.487142    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998e310e-0a23-49a3-bc01-f5c827b20c08" path="/var/lib/kubelet/pods/998e310e-0a23-49a3-bc01-f5c827b20c08/volumes"
Sep 18 14:10:30.241895 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:30.241859    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"]
Sep 18 14:10:30.242353 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:30.242155    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy" containerID="cri-o://cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc" gracePeriod=30
Sep 18 14:10:35.489222 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.489200    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:10:35.677768 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.677674    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c0ae47-d6db-4759-96e8-1c803ddedf94-istiod-ca-cert\") pod \"95c0ae47-d6db-4759-96e8-1c803ddedf94\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") "
Sep 18 14:10:35.677768 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.677711    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-certs\") pod \"95c0ae47-d6db-4759-96e8-1c803ddedf94\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") "
Sep 18 14:10:35.677768 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.677738    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-socket\") pod \"95c0ae47-d6db-4759-96e8-1c803ddedf94\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") "
Sep 18 14:10:35.677768 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.677764    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-envoy\") pod \"95c0ae47-d6db-4759-96e8-1c803ddedf94\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") "
Sep 18 14:10:35.678109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.677800    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-podinfo\") pod \"95c0ae47-d6db-4759-96e8-1c803ddedf94\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") "
Sep 18 14:10:35.678109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.677817    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-data\") pod \"95c0ae47-d6db-4759-96e8-1c803ddedf94\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") "
Sep 18 14:10:35.678109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.677838    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-token\") pod \"95c0ae47-d6db-4759-96e8-1c803ddedf94\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") "
Sep 18 14:10:35.678109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.677887    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgc8b\" (UniqueName: \"kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-kube-api-access-lgc8b\") pod \"95c0ae47-d6db-4759-96e8-1c803ddedf94\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") "
Sep 18 14:10:35.678109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.677928    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-credential-socket\") pod \"95c0ae47-d6db-4759-96e8-1c803ddedf94\" (UID: \"95c0ae47-d6db-4759-96e8-1c803ddedf94\") "
Sep 18 14:10:35.678109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.677971    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95c0ae47-d6db-4759-96e8-1c803ddedf94" (UID: "95c0ae47-d6db-4759-96e8-1c803ddedf94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:35.678427 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.678284    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:35.678427 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.678281    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-data" (OuterVolumeSpecName: "istio-data") pod "95c0ae47-d6db-4759-96e8-1c803ddedf94" (UID: "95c0ae47-d6db-4759-96e8-1c803ddedf94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:35.678427 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.678297    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95c0ae47-d6db-4759-96e8-1c803ddedf94" (UID: "95c0ae47-d6db-4759-96e8-1c803ddedf94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:35.678427 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.678306    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c0ae47-d6db-4759-96e8-1c803ddedf94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95c0ae47-d6db-4759-96e8-1c803ddedf94" (UID: "95c0ae47-d6db-4759-96e8-1c803ddedf94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:10:35.678626 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.678536    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95c0ae47-d6db-4759-96e8-1c803ddedf94" (UID: "95c0ae47-d6db-4759-96e8-1c803ddedf94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:35.680252 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.680226    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95c0ae47-d6db-4759-96e8-1c803ddedf94" (UID: "95c0ae47-d6db-4759-96e8-1c803ddedf94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:10:35.680631 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.680611    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-kube-api-access-lgc8b" (OuterVolumeSpecName: "kube-api-access-lgc8b") pod "95c0ae47-d6db-4759-96e8-1c803ddedf94" (UID: "95c0ae47-d6db-4759-96e8-1c803ddedf94"). InnerVolumeSpecName "kube-api-access-lgc8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:35.680776 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.680748    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95c0ae47-d6db-4759-96e8-1c803ddedf94" (UID: "95c0ae47-d6db-4759-96e8-1c803ddedf94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:10:35.680776 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.680753    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-token" (OuterVolumeSpecName: "istio-token") pod "95c0ae47-d6db-4759-96e8-1c803ddedf94" (UID: "95c0ae47-d6db-4759-96e8-1c803ddedf94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:35.779499 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.779463    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-data\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:35.779499 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.779495    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-token\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:35.779499 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.779505    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgc8b\" (UniqueName: \"kubernetes.io/projected/95c0ae47-d6db-4759-96e8-1c803ddedf94-kube-api-access-lgc8b\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:35.779700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.779516    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-credential-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:35.779700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.779525    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95c0ae47-d6db-4759-96e8-1c803ddedf94-istiod-ca-cert\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:35.779700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.779532    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-workload-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:35.779700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.779540    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-envoy\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:35.779700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:35.779547    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95c0ae47-d6db-4759-96e8-1c803ddedf94-istio-podinfo\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:36.111372 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.111342    2569 generic.go:356] "Generic (PLEG): container finished" podID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerID="cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc" exitCode=0
Sep 18 14:10:36.111531 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.111390    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"
Sep 18 14:10:36.111531 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.111419    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" event={"ID":"95c0ae47-d6db-4759-96e8-1c803ddedf94","Type":"ContainerDied","Data":"cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc"}
Sep 18 14:10:36.111531 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.111475    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz" event={"ID":"95c0ae47-d6db-4759-96e8-1c803ddedf94","Type":"ContainerDied","Data":"5b101c5b36ac6d24570ea79dc94d57c70c7679f5e857800f976a806ae4b6a3bc"}
Sep 18 14:10:36.111531 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.111497    2569 scope.go:117] "RemoveContainer" containerID="cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc"
Sep 18 14:10:36.120832 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.120816    2569 scope.go:117] "RemoveContainer" containerID="b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816"
Sep 18 14:10:36.129429 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.129412    2569 scope.go:117] "RemoveContainer" containerID="cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc"
Sep 18 14:10:36.129688 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:10:36.129669    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc\": container with ID starting with cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc not found: ID does not exist" containerID="cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc"
Sep 18 14:10:36.129749 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.129696    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc"} err="failed to get container status \"cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc\": rpc error: code = NotFound desc = could not find container \"cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc\": container with ID starting with cc28ef75b33c2b37e32db1d1b0ad10be7f3bd93f1ade3a7de1ccbd461bb13dfc not found: ID does not exist"
Sep 18 14:10:36.129749 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.129713    2569 scope.go:117] "RemoveContainer" containerID="b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816"
Sep 18 14:10:36.129964 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:10:36.129948    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816\": container with ID starting with b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816 not found: ID does not exist" containerID="b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816"
Sep 18 14:10:36.130020 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.129972    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816"} err="failed to get container status \"b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816\": rpc error: code = NotFound desc = could not find container \"b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816\": container with ID starting with b0a3bba2281186132ff7e880bb5dfcb8d5d348d4fc5ff08057d4d8588c87a816 not found: ID does not exist"
Sep 18 14:10:36.141504 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.141476    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"]
Sep 18 14:10:36.142486 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.142468    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e46fbf-openshift-default-7d85569b9b-z5snz"]
Sep 18 14:10:36.485899 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:36.485868    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" path="/var/lib/kubelet/pods/95c0ae47-d6db-4759-96e8-1c803ddedf94/volumes"
Sep 18 14:10:41.041717 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.041687    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"]
Sep 18 14:10:41.042213 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.041933    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc" podUID="b6941922-7692-4ac6-b5ea-fe1139735fb9" containerName="maas-api" containerID="cri-o://0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364" gracePeriod=30
Sep 18 14:10:41.097694 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.097665    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"]
Sep 18 14:10:41.097894 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.097867    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs" podUID="937777ed-724d-4b69-865d-0e9e64d05aaf" containerName="payload-processing" containerID="cri-o://08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5" gracePeriod=30
Sep 18 14:10:41.356096 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.355808    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:10:41.391259 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.391234    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:10:41.426518 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.426492    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-maas-api-tls\") pod \"b6941922-7692-4ac6-b5ea-fe1139735fb9\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") "
Sep 18 14:10:41.426676 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.426540    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-metrics-certs\") pod \"b6941922-7692-4ac6-b5ea-fe1139735fb9\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") "
Sep 18 14:10:41.426676 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.426653    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjmnm\" (UniqueName: \"kubernetes.io/projected/937777ed-724d-4b69-865d-0e9e64d05aaf-kube-api-access-pjmnm\") pod \"937777ed-724d-4b69-865d-0e9e64d05aaf\" (UID: \"937777ed-724d-4b69-865d-0e9e64d05aaf\") "
Sep 18 14:10:41.426811 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.426680    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhphp\" (UniqueName: \"kubernetes.io/projected/b6941922-7692-4ac6-b5ea-fe1139735fb9-kube-api-access-rhphp\") pod \"b6941922-7692-4ac6-b5ea-fe1139735fb9\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") "
Sep 18 14:10:41.426811 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.426713    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937777ed-724d-4b69-865d-0e9e64d05aaf-plugins-config-volume\") pod \"937777ed-724d-4b69-865d-0e9e64d05aaf\" (UID: \"937777ed-724d-4b69-865d-0e9e64d05aaf\") "
Sep 18 14:10:41.426811 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.426740    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6941922-7692-4ac6-b5ea-fe1139735fb9-openshift-service-ca\") pod \"b6941922-7692-4ac6-b5ea-fe1139735fb9\" (UID: \"b6941922-7692-4ac6-b5ea-fe1139735fb9\") "
Sep 18 14:10:41.428288 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.427580    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937777ed-724d-4b69-865d-0e9e64d05aaf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "937777ed-724d-4b69-865d-0e9e64d05aaf" (UID: "937777ed-724d-4b69-865d-0e9e64d05aaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:10:41.430221 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.429926    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b6941922-7692-4ac6-b5ea-fe1139735fb9" (UID: "b6941922-7692-4ac6-b5ea-fe1139735fb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:10:41.430221 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.429967    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6941922-7692-4ac6-b5ea-fe1139735fb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b6941922-7692-4ac6-b5ea-fe1139735fb9" (UID: "b6941922-7692-4ac6-b5ea-fe1139735fb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:10:41.430473 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.430449    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937777ed-724d-4b69-865d-0e9e64d05aaf-kube-api-access-pjmnm" (OuterVolumeSpecName: "kube-api-access-pjmnm") pod "937777ed-724d-4b69-865d-0e9e64d05aaf" (UID: "937777ed-724d-4b69-865d-0e9e64d05aaf"). InnerVolumeSpecName "kube-api-access-pjmnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:41.431481 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.431453    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6941922-7692-4ac6-b5ea-fe1139735fb9-kube-api-access-rhphp" (OuterVolumeSpecName: "kube-api-access-rhphp") pod "b6941922-7692-4ac6-b5ea-fe1139735fb9" (UID: "b6941922-7692-4ac6-b5ea-fe1139735fb9"). InnerVolumeSpecName "kube-api-access-rhphp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:41.436879 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.436430    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b6941922-7692-4ac6-b5ea-fe1139735fb9" (UID: "b6941922-7692-4ac6-b5ea-fe1139735fb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:10:41.528521 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.528491    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjmnm\" (UniqueName: \"kubernetes.io/projected/937777ed-724d-4b69-865d-0e9e64d05aaf-kube-api-access-pjmnm\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:41.528521 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.528515    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhphp\" (UniqueName: \"kubernetes.io/projected/b6941922-7692-4ac6-b5ea-fe1139735fb9-kube-api-access-rhphp\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:41.528734 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.528533    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937777ed-724d-4b69-865d-0e9e64d05aaf-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:41.528734 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.528542    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6941922-7692-4ac6-b5ea-fe1139735fb9-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:41.528734 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.528551    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:41.528734 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:41.528560    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6941922-7692-4ac6-b5ea-fe1139735fb9-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:10:42.142183 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.142142    2569 generic.go:356] "Generic (PLEG): container finished" podID="b6941922-7692-4ac6-b5ea-fe1139735fb9" containerID="0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364" exitCode=0
Sep 18 14:10:42.142641 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.142202    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc" event={"ID":"b6941922-7692-4ac6-b5ea-fe1139735fb9","Type":"ContainerDied","Data":"0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364"}
Sep 18 14:10:42.142641 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.142269    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc" event={"ID":"b6941922-7692-4ac6-b5ea-fe1139735fb9","Type":"ContainerDied","Data":"45f6d115b1d23a4b3a198b2b8b64cb8485587ee9a735b9c9e2143cc4ae6936dc"}
Sep 18 14:10:42.142641 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.142210    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"
Sep 18 14:10:42.142641 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.142302    2569 scope.go:117] "RemoveContainer" containerID="0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364"
Sep 18 14:10:42.143794 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.143656    2569 generic.go:356] "Generic (PLEG): container finished" podID="937777ed-724d-4b69-865d-0e9e64d05aaf" containerID="08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5" exitCode=0
Sep 18 14:10:42.143794 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.143703    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"
Sep 18 14:10:42.143794 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.143702    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs" event={"ID":"937777ed-724d-4b69-865d-0e9e64d05aaf","Type":"ContainerDied","Data":"08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5"}
Sep 18 14:10:42.143794 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.143760    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs" event={"ID":"937777ed-724d-4b69-865d-0e9e64d05aaf","Type":"ContainerDied","Data":"255debc167772a1307e1209886a38b5f64fd4f047be3861000660958e02ab65b"}
Sep 18 14:10:42.152738 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.152723    2569 scope.go:117] "RemoveContainer" containerID="0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364"
Sep 18 14:10:42.153113 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:10:42.153038    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364\": container with ID starting with 0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364 not found: ID does not exist" containerID="0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364"
Sep 18 14:10:42.153113 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.153084    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364"} err="failed to get container status \"0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364\": rpc error: code = NotFound desc = could not find container \"0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364\": container with ID starting with 0eca5d8d690640b9fc8ea1f78dcbfe6f62618de07cf46836ac8679382add8364 not found: ID does not exist"
Sep 18 14:10:42.153113 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.153103    2569 scope.go:117] "RemoveContainer" containerID="08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5"
Sep 18 14:10:42.161112 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.161096    2569 scope.go:117] "RemoveContainer" containerID="08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5"
Sep 18 14:10:42.161343 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:10:42.161328    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5\": container with ID starting with 08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5 not found: ID does not exist" containerID="08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5"
Sep 18 14:10:42.161411 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.161347    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5"} err="failed to get container status \"08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5\": rpc error: code = NotFound desc = could not find container \"08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5\": container with ID starting with 08b1b6fdc360411a788f821f0e9a295e586947a1ead4de7420b2c1a64fa35eb5 not found: ID does not exist"
Sep 18 14:10:42.168727 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.168707    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"]
Sep 18 14:10:42.174955 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.174935    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-55bd4d-94f6db896-jl9qc"]
Sep 18 14:10:42.187312 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.187274    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"]
Sep 18 14:10:42.189134 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.189113    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55bd4d-5f9d956479-qfgcs"]
Sep 18 14:10:42.485659 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.485623    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="937777ed-724d-4b69-865d-0e9e64d05aaf" path="/var/lib/kubelet/pods/937777ed-724d-4b69-865d-0e9e64d05aaf/volumes"
Sep 18 14:10:42.486118 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:10:42.486099    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6941922-7692-4ac6-b5ea-fe1139735fb9" path="/var/lib/kubelet/pods/b6941922-7692-4ac6-b5ea-fe1139735fb9/volumes"
Sep 18 14:11:07.165898 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.165865    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7627-bf2a35"]
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166233    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166245    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166257    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed1c0dd6-c519-4bc5-8b09-c7d30399073d" containerName="maas-api"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166262    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed1c0dd6-c519-4bc5-8b09-c7d30399073d" containerName="maas-api"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166270    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2848a5b6-e7ed-4fbc-a750-556cfea2a76a" containerName="payload-processing"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166276    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2848a5b6-e7ed-4fbc-a750-556cfea2a76a" containerName="payload-processing"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166283    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c48bddc-6e45-430f-97be-753cf5efd021" containerName="payload-pre-processing"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166288    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c48bddc-6e45-430f-97be-753cf5efd021" containerName="payload-pre-processing"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166294    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998e310e-0a23-49a3-bc01-f5c827b20c08" containerName="maas-api"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166299    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="998e310e-0a23-49a3-bc01-f5c827b20c08" containerName="maas-api"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166309    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166314    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166322    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166327    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166333    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80282ae-09d3-42b0-9cfd-d1495199f99f" containerName="main"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166338    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80282ae-09d3-42b0-9cfd-d1495199f99f" containerName="main"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166345    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6941922-7692-4ac6-b5ea-fe1139735fb9" containerName="maas-api"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166349    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6941922-7692-4ac6-b5ea-fe1139735fb9" containerName="maas-api"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166356    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937777ed-724d-4b69-865d-0e9e64d05aaf" containerName="payload-processing"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166361    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="937777ed-724d-4b69-865d-0e9e64d05aaf" containerName="payload-processing"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166370    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55445413-b746-48fd-b78b-f58aaee8a1e1" containerName="maas-api"
Sep 18 14:11:07.166363 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166374    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55445413-b746-48fd-b78b-f58aaee8a1e1" containerName="maas-api"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166428    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166434    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c0ae47-d6db-4759-96e8-1c803ddedf94" containerName="istio-proxy"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166439    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="937777ed-724d-4b69-865d-0e9e64d05aaf" containerName="payload-processing"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166445    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed1c0dd6-c519-4bc5-8b09-c7d30399073d" containerName="maas-api"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166452    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80282ae-09d3-42b0-9cfd-d1495199f99f" containerName="main"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166458    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2848a5b6-e7ed-4fbc-a750-556cfea2a76a" containerName="payload-processing"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166463    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c48bddc-6e45-430f-97be-753cf5efd021" containerName="payload-pre-processing"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166469    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166474    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6941922-7692-4ac6-b5ea-fe1139735fb9" containerName="maas-api"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166479    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="55445413-b746-48fd-b78b-f58aaee8a1e1" containerName="maas-api"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166483    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy"
Sep 18 14:11:07.167132 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.166488    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="998e310e-0a23-49a3-bc01-f5c827b20c08" containerName="maas-api"
Sep 18 14:11:07.168566 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.168551    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-bf2a35"
Sep 18 14:11:07.171492 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.171472    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-knszx\""
Sep 18 14:11:07.175850 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.175707    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-bf2a35"]
Sep 18 14:11:07.260929 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.260894    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnsvf\" (UniqueName: \"kubernetes.io/projected/646fb0c2-3f8f-4609-8f22-d5ddb77aae84-kube-api-access-nnsvf\") pod \"test-curl-7627-bf2a35\" (UID: \"646fb0c2-3f8f-4609-8f22-d5ddb77aae84\") " pod="opendatahub/test-curl-7627-bf2a35"
Sep 18 14:11:07.362423 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.362389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsvf\" (UniqueName: \"kubernetes.io/projected/646fb0c2-3f8f-4609-8f22-d5ddb77aae84-kube-api-access-nnsvf\") pod \"test-curl-7627-bf2a35\" (UID: \"646fb0c2-3f8f-4609-8f22-d5ddb77aae84\") " pod="opendatahub/test-curl-7627-bf2a35"
Sep 18 14:11:07.371343 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.371317    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsvf\" (UniqueName: \"kubernetes.io/projected/646fb0c2-3f8f-4609-8f22-d5ddb77aae84-kube-api-access-nnsvf\") pod \"test-curl-7627-bf2a35\" (UID: \"646fb0c2-3f8f-4609-8f22-d5ddb77aae84\") " pod="opendatahub/test-curl-7627-bf2a35"
Sep 18 14:11:07.478767 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.478739    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-bf2a35"
Sep 18 14:11:07.747042 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:07.747016    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-bf2a35"]
Sep 18 14:11:07.749485 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:11:07.749452    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod646fb0c2_3f8f_4609_8f22_d5ddb77aae84.slice/crio-2ddfbe937be0bed1a4d6e701a0411b1e336e6b664603b322f8772d81c8292505 WatchSource:0}: Error finding container 2ddfbe937be0bed1a4d6e701a0411b1e336e6b664603b322f8772d81c8292505: Status 404 returned error can't find the container with id 2ddfbe937be0bed1a4d6e701a0411b1e336e6b664603b322f8772d81c8292505
Sep 18 14:11:08.268787 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:08.268753    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-bf2a35" event={"ID":"646fb0c2-3f8f-4609-8f22-d5ddb77aae84","Type":"ContainerStarted","Data":"2ddfbe937be0bed1a4d6e701a0411b1e336e6b664603b322f8772d81c8292505"}
Sep 18 14:11:10.280779 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:10.280740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-bf2a35" event={"ID":"646fb0c2-3f8f-4609-8f22-d5ddb77aae84","Type":"ContainerStarted","Data":"1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94"}
Sep 18 14:11:10.298133 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:10.298058    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7627-bf2a35" podStartSLOduration=1.334998281 podStartE2EDuration="3.29804133s" totalImagesPullingTime="1.963043049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:10.296893291 +0000 UTC m=+1612.199349477" watchObservedRunningTime="2026-09-18 14:11:10.29804133 +0000 UTC m=+1612.200497515"
Sep 18 14:11:10.884767 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:10.884729    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7627-bf2a35"]
Sep 18 14:11:10.889281 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:10.889255    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7627-bf2a35"]
Sep 18 14:11:11.011308 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.011274    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7627-67088d"]
Sep 18 14:11:11.011694 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.011677    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy"
Sep 18 14:11:11.011781 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.011697    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bd4b0fb-8d52-4a32-a447-9b65c7d43cf5" containerName="istio-proxy"
Sep 18 14:11:11.011781 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.011716    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646fb0c2-3f8f-4609-8f22-d5ddb77aae84" containerName="test-curl-7627-bf2a35"
Sep 18 14:11:11.011781 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.011724    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="646fb0c2-3f8f-4609-8f22-d5ddb77aae84" containerName="test-curl-7627-bf2a35"
Sep 18 14:11:11.011934 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.011803    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="646fb0c2-3f8f-4609-8f22-d5ddb77aae84" containerName="test-curl-7627-bf2a35"
Sep 18 14:11:11.013782 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.013763    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:11.021418 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.021394    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-67088d"]
Sep 18 14:11:11.197006 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.196969    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbk2\" (UniqueName: \"kubernetes.io/projected/d9d80f2d-e75a-47ae-9389-d05539f75cd9-kube-api-access-8bbk2\") pod \"test-curl-7627-67088d\" (UID: \"d9d80f2d-e75a-47ae-9389-d05539f75cd9\") " pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:11.285344 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:11:11.285267    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7627-bf2a35" container="test-curl-7627-bf2a35"
Sep 18 14:11:11.297450 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.297430    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbk2\" (UniqueName: \"kubernetes.io/projected/d9d80f2d-e75a-47ae-9389-d05539f75cd9-kube-api-access-8bbk2\") pod \"test-curl-7627-67088d\" (UID: \"d9d80f2d-e75a-47ae-9389-d05539f75cd9\") " pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:11.306724 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.306698    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbk2\" (UniqueName: \"kubernetes.io/projected/d9d80f2d-e75a-47ae-9389-d05539f75cd9-kube-api-access-8bbk2\") pod \"test-curl-7627-67088d\" (UID: \"d9d80f2d-e75a-47ae-9389-d05539f75cd9\") " pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:11.322842 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.322820    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:11.599864 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:11.599706    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-67088d"]
Sep 18 14:11:11.602551 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:11:11.602528    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9d80f2d_e75a_47ae_9389_d05539f75cd9.slice/crio-f30d921281c5e1aaa2151038a519f463c90f392cecab3fe30126225e5b6c237d WatchSource:0}: Error finding container f30d921281c5e1aaa2151038a519f463c90f392cecab3fe30126225e5b6c237d: Status 404 returned error can't find the container with id f30d921281c5e1aaa2151038a519f463c90f392cecab3fe30126225e5b6c237d
Sep 18 14:11:12.291123 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:12.291086    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-67088d" event={"ID":"d9d80f2d-e75a-47ae-9389-d05539f75cd9","Type":"ContainerStarted","Data":"5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53"}
Sep 18 14:11:12.291123 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:12.291128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-67088d" event={"ID":"d9d80f2d-e75a-47ae-9389-d05539f75cd9","Type":"ContainerStarted","Data":"f30d921281c5e1aaa2151038a519f463c90f392cecab3fe30126225e5b6c237d"}
Sep 18 14:11:12.291652 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:12.291225    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7627-bf2a35" podUID="646fb0c2-3f8f-4609-8f22-d5ddb77aae84" containerName="test-curl-7627-bf2a35" containerID="cri-o://1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94" gracePeriod=2
Sep 18 14:11:12.307366 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:12.307318    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7627-67088d" podStartSLOduration=2.307299717 podStartE2EDuration="2.307299717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:12.306552299 +0000 UTC m=+1614.209008486" watchObservedRunningTime="2026-09-18 14:11:12.307299717 +0000 UTC m=+1614.209755906"
Sep 18 14:11:12.309911 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:11:12.309889    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7627-bf2a35\" is forbidden: User \"system:node:ip-10-0-143-135.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-135.ec2.internal' and this object" podUID="646fb0c2-3f8f-4609-8f22-d5ddb77aae84" pod="opendatahub/test-curl-7627-bf2a35"
Sep 18 14:11:12.939630 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:12.939595    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7627-67088d"]
Sep 18 14:11:12.943700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:12.943680    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7627-67088d"]
Sep 18 14:11:13.295829 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:11:13.295798    2569 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7627-67088d" container="test-curl-7627-67088d"
Sep 18 14:11:14.300678 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:14.300644    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7627-67088d" podUID="d9d80f2d-e75a-47ae-9389-d05539f75cd9" containerName="test-curl-7627-67088d" containerID="cri-o://5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53" gracePeriod=2
Sep 18 14:11:14.303379 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:11:14.303353    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7627-67088d\" is forbidden: User \"system:node:ip-10-0-143-135.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-135.ec2.internal' and this object" podUID="d9d80f2d-e75a-47ae-9389-d05539f75cd9" pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:14.478345 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:14.478322    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-bf2a35"
Sep 18 14:11:14.480950 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:11:14.480922    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7627-67088d\" is forbidden: User \"system:node:ip-10-0-143-135.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-135.ec2.internal' and this object" podUID="d9d80f2d-e75a-47ae-9389-d05539f75cd9" pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:14.629536 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:14.629445    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnsvf\" (UniqueName: \"kubernetes.io/projected/646fb0c2-3f8f-4609-8f22-d5ddb77aae84-kube-api-access-nnsvf\") pod \"646fb0c2-3f8f-4609-8f22-d5ddb77aae84\" (UID: \"646fb0c2-3f8f-4609-8f22-d5ddb77aae84\") "
Sep 18 14:11:14.631793 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:14.631762    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646fb0c2-3f8f-4609-8f22-d5ddb77aae84-kube-api-access-nnsvf" (OuterVolumeSpecName: "kube-api-access-nnsvf") pod "646fb0c2-3f8f-4609-8f22-d5ddb77aae84" (UID: "646fb0c2-3f8f-4609-8f22-d5ddb77aae84"). InnerVolumeSpecName "kube-api-access-nnsvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:11:14.730156 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:14.730121    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnsvf\" (UniqueName: \"kubernetes.io/projected/646fb0c2-3f8f-4609-8f22-d5ddb77aae84-kube-api-access-nnsvf\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:11:15.312166 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:15.312121    2569 generic.go:356] "Generic (PLEG): container finished" podID="646fb0c2-3f8f-4609-8f22-d5ddb77aae84" containerID="1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94" exitCode=137
Sep 18 14:11:15.312634 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:15.312181    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-bf2a35"
Sep 18 14:11:15.312634 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:15.312220    2569 scope.go:117] "RemoveContainer" containerID="1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94"
Sep 18 14:11:15.321148 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:15.321129    2569 scope.go:117] "RemoveContainer" containerID="1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94"
Sep 18 14:11:15.321427 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:11:15.321409    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94\": container with ID starting with 1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94 not found: ID does not exist" containerID="1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94"
Sep 18 14:11:15.321479 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:15.321435    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94"} err="failed to get container status \"1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94\": rpc error: code = NotFound desc = could not find container \"1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94\": container with ID starting with 1df390627cae129b0b45cb3e5d762d9b8703eedd54fb6043bc47f80789ad6b94 not found: ID does not exist"
Sep 18 14:11:16.485850 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:16.485819    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646fb0c2-3f8f-4609-8f22-d5ddb77aae84" path="/var/lib/kubelet/pods/646fb0c2-3f8f-4609-8f22-d5ddb77aae84/volumes"
Sep 18 14:11:16.486429 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:16.486412    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:16.650973 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:16.650886    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bbk2\" (UniqueName: \"kubernetes.io/projected/d9d80f2d-e75a-47ae-9389-d05539f75cd9-kube-api-access-8bbk2\") pod \"d9d80f2d-e75a-47ae-9389-d05539f75cd9\" (UID: \"d9d80f2d-e75a-47ae-9389-d05539f75cd9\") "
Sep 18 14:11:16.653211 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:16.653186    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d80f2d-e75a-47ae-9389-d05539f75cd9-kube-api-access-8bbk2" (OuterVolumeSpecName: "kube-api-access-8bbk2") pod "d9d80f2d-e75a-47ae-9389-d05539f75cd9" (UID: "d9d80f2d-e75a-47ae-9389-d05539f75cd9"). InnerVolumeSpecName "kube-api-access-8bbk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:11:16.751713 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:16.751684    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbk2\" (UniqueName: \"kubernetes.io/projected/d9d80f2d-e75a-47ae-9389-d05539f75cd9-kube-api-access-8bbk2\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:11:17.324505 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:17.324471    2569 generic.go:356] "Generic (PLEG): container finished" podID="d9d80f2d-e75a-47ae-9389-d05539f75cd9" containerID="5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53" exitCode=137
Sep 18 14:11:17.324683 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:17.324514    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:17.324683 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:17.324522    2569 scope.go:117] "RemoveContainer" containerID="5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53"
Sep 18 14:11:17.333037 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:17.333014    2569 scope.go:117] "RemoveContainer" containerID="5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53"
Sep 18 14:11:17.333311 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:11:17.333293    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53\": container with ID starting with 5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53 not found: ID does not exist" containerID="5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53"
Sep 18 14:11:17.333370 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:17.333321    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53"} err="failed to get container status \"5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53\": rpc error: code = NotFound desc = could not find container \"5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53\": container with ID starting with 5678c23aafd5374514759580e8a97f20a3d7319d6bf54a536898a62dc006ab53 not found: ID does not exist"
Sep 18 14:11:17.335130 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:11:17.335109    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7627-67088d\" is forbidden: User \"system:node:ip-10-0-143-135.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-135.ec2.internal' and this object" podUID="d9d80f2d-e75a-47ae-9389-d05539f75cd9" pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:18.484973 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:11:18.484941    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7627-67088d\" is forbidden: User \"system:node:ip-10-0-143-135.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-135.ec2.internal' and this object" podUID="d9d80f2d-e75a-47ae-9389-d05539f75cd9" pod="opendatahub/test-curl-7627-67088d"
Sep 18 14:11:18.486019 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:18.485997    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d80f2d-e75a-47ae-9389-d05539f75cd9" path="/var/lib/kubelet/pods/d9d80f2d-e75a-47ae-9389-d05539f75cd9/volumes"
Sep 18 14:11:23.213143 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.213109    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"]
Sep 18 14:11:23.213621 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.213604    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d80f2d-e75a-47ae-9389-d05539f75cd9" containerName="test-curl-7627-67088d"
Sep 18 14:11:23.213720 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.213626    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d80f2d-e75a-47ae-9389-d05539f75cd9" containerName="test-curl-7627-67088d"
Sep 18 14:11:23.213720 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.213708    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d80f2d-e75a-47ae-9389-d05539f75cd9" containerName="test-curl-7627-67088d"
Sep 18 14:11:23.215680 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.215660    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.220268 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.220247    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-783872\"/\"rate-test-model-783872-kserve-self-signed-certs\""
Sep 18 14:11:23.221467 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.221438    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-783872\"/\"openshift-service-ca.crt\""
Sep 18 14:11:23.221580 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.221530    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-783872\"/\"default-dockercfg-cwdxt\""
Sep 18 14:11:23.221652 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.221598    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-783872\"/\"kube-root-ca.crt\""
Sep 18 14:11:23.229644 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.229621    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"]
Sep 18 14:11:23.299379 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.299348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-tmp-dir\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.299529 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.299391    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61191bdf-ae6a-4377-b969-9ebd1cbca893-tls-certs\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.299529 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.299435    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-model-cache\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.299529 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.299455    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmdm\" (UniqueName: \"kubernetes.io/projected/61191bdf-ae6a-4377-b969-9ebd1cbca893-kube-api-access-kxmdm\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.299529 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.299476    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-home\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.299685 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.299560    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-dshm\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.400733 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.400705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-dshm\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.400945 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.400750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-tmp-dir\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.400945 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.400884    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61191bdf-ae6a-4377-b969-9ebd1cbca893-tls-certs\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.400945 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.400925    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-model-cache\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.401158 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.400957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmdm\" (UniqueName: \"kubernetes.io/projected/61191bdf-ae6a-4377-b969-9ebd1cbca893-kube-api-access-kxmdm\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.401158 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.400995    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-home\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.401158 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.401148    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-tmp-dir\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.401346 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.401226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-model-cache\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.401414 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.401375    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-home\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.403253 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.403225    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-dshm\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.403484 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.403468    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61191bdf-ae6a-4377-b969-9ebd1cbca893-tls-certs\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.410506 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.410471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmdm\" (UniqueName: \"kubernetes.io/projected/61191bdf-ae6a-4377-b969-9ebd1cbca893-kube-api-access-kxmdm\") pod \"rate-test-model-783872-kserve-559ddf5779-59r25\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") " pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.525993 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.525884    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:23.803157 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:23.803132    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"]
Sep 18 14:11:23.805357 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:11:23.805322    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61191bdf_ae6a_4377_b969_9ebd1cbca893.slice/crio-3837dfcfcf60163e420ddfb75fbc14c1649bdbee26bb972addb3a773b39fee10 WatchSource:0}: Error finding container 3837dfcfcf60163e420ddfb75fbc14c1649bdbee26bb972addb3a773b39fee10: Status 404 returned error can't find the container with id 3837dfcfcf60163e420ddfb75fbc14c1649bdbee26bb972addb3a773b39fee10
Sep 18 14:11:24.361274 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:24.361240    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25" event={"ID":"61191bdf-ae6a-4377-b969-9ebd1cbca893","Type":"ContainerStarted","Data":"cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237"}
Sep 18 14:11:24.361274 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:24.361278    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25" event={"ID":"61191bdf-ae6a-4377-b969-9ebd1cbca893","Type":"ContainerStarted","Data":"3837dfcfcf60163e420ddfb75fbc14c1649bdbee26bb972addb3a773b39fee10"}
Sep 18 14:11:24.380489 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:24.380442    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25" podStartSLOduration=1.380426847 podStartE2EDuration="1.380426847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:24.379281898 +0000 UTC m=+1626.281738097" watchObservedRunningTime="2026-09-18 14:11:24.380426847 +0000 UTC m=+1626.282883033"
Sep 18 14:11:33.526826 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:33.526793    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:33.527348 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:33.526842    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:33.539196 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:33.539165    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:11:34.423740 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:11:34.423706    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:12:29.640358 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:29.640317    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"]
Sep 18 14:12:31.438973 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.438940    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"]
Sep 18 14:12:31.441586 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.441571    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.444451 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.444433    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b6289c-openshift-default-dockercfg-pxzln\""
Sep 18 14:12:31.457005 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.456981    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"]
Sep 18 14:12:31.496609 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.496576    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9s55\" (UniqueName: \"kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-kube-api-access-t9s55\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.496749 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.496624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-credential-socket\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.496749 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.496692    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-socket\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.496749 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.496730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-istio-token\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.496860 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.496756    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43998e37-a725-46d1-881c-721ac526aff2-istiod-ca-cert\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.496860 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.496840    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-certs\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.496926 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.496862    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43998e37-a725-46d1-881c-721ac526aff2-istio-podinfo\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.496926 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.496882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-envoy\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.496926 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.496906    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-data\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.597517 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597483    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-envoy\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.597517 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-data\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.597774 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9s55\" (UniqueName: \"kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-kube-api-access-t9s55\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.597774 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-credential-socket\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.597774 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597633    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-socket\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.597941 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597786    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-istio-token\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.597941 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597846    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43998e37-a725-46d1-881c-721ac526aff2-istiod-ca-cert\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.597941 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-certs\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.597941 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597934    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43998e37-a725-46d1-881c-721ac526aff2-istio-podinfo\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.598174 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597966    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-socket\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.598174 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.597984    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-data\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.598404 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.598379    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-credential-socket\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.598477 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.598457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-certs\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.598600 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.598584    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43998e37-a725-46d1-881c-721ac526aff2-istiod-ca-cert\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.599999 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.599982    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-envoy\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.600408 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.600389    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43998e37-a725-46d1-881c-721ac526aff2-istio-podinfo\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.607671 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.607562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9s55\" (UniqueName: \"kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-kube-api-access-t9s55\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.607822 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.607713    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-istio-token\") pod \"e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") " pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:31.751434 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:31.751399    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:32.033682 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:32.033657    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"]
Sep 18 14:12:32.035624 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:12:32.035595    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43998e37_a725_46d1_881c_721ac526aff2.slice/crio-dab8ad59b5e6a6f0e50a90fac4c3688aedbd50d1a6069ebf16b44109dca63fc6 WatchSource:0}: Error finding container dab8ad59b5e6a6f0e50a90fac4c3688aedbd50d1a6069ebf16b44109dca63fc6: Status 404 returned error can't find the container with id dab8ad59b5e6a6f0e50a90fac4c3688aedbd50d1a6069ebf16b44109dca63fc6
Sep 18 14:12:32.037808 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:32.037776    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:12:32.037903 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:32.037842    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:12:32.037903 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:32.037868    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:12:32.684989 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:32.684952    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" event={"ID":"43998e37-a725-46d1-881c-721ac526aff2","Type":"ContainerStarted","Data":"777c21aa3b7caf7325db41e1b58496154f4a4a45f3c04f0bbfe74bfc5bb55297"}
Sep 18 14:12:32.685398 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:32.685008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" event={"ID":"43998e37-a725-46d1-881c-721ac526aff2","Type":"ContainerStarted","Data":"dab8ad59b5e6a6f0e50a90fac4c3688aedbd50d1a6069ebf16b44109dca63fc6"}
Sep 18 14:12:32.710555 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:32.710510    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" podStartSLOduration=1.710493708 podStartE2EDuration="1.710493708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:12:32.708976083 +0000 UTC m=+1694.611432268" watchObservedRunningTime="2026-09-18 14:12:32.710493708 +0000 UTC m=+1694.612949894"
Sep 18 14:12:32.752232 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:32.752204    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:33.757889 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:33.757859    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:34.694562 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:34.694533    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:34.695568 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:34.695543    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:36.783500 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:36.783469    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"]
Sep 18 14:12:36.786660 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:36.786644    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:36.790316 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:36.790298    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b6289c-serving-cert\""
Sep 18 14:12:36.807620 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:36.807600    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"]
Sep 18 14:12:36.947139 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:36.947111    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7hth\" (UniqueName: \"kubernetes.io/projected/5f1fcb61-15a1-493b-af21-6198786c17bd-kube-api-access-w7hth\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:36.947356 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:36.947225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-metrics-certs\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:36.947356 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:36.947261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:36.947356 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:36.947294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1fcb61-15a1-493b-af21-6198786c17bd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:37.048325 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.048241    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-metrics-certs\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:37.048325 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.048281    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:37.048325 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.048305    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1fcb61-15a1-493b-af21-6198786c17bd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:37.048609 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.048383    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7hth\" (UniqueName: \"kubernetes.io/projected/5f1fcb61-15a1-493b-af21-6198786c17bd-kube-api-access-w7hth\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:37.049051 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.049029    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1fcb61-15a1-493b-af21-6198786c17bd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:37.050902 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.050878    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-metrics-certs\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:37.051083 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.051047    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:37.065449 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.065420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7hth\" (UniqueName: \"kubernetes.io/projected/5f1fcb61-15a1-493b-af21-6198786c17bd-kube-api-access-w7hth\") pod \"maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:37.097509 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.097482    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:37.378814 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.378791    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"]
Sep 18 14:12:37.381151 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:12:37.381120    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1fcb61_15a1_493b_af21_6198786c17bd.slice/crio-2c9ab5dda2023ccdd1d17655b1bcc22e5beba0cb71468df1c6b2c184a1777904 WatchSource:0}: Error finding container 2c9ab5dda2023ccdd1d17655b1bcc22e5beba0cb71468df1c6b2c184a1777904: Status 404 returned error can't find the container with id 2c9ab5dda2023ccdd1d17655b1bcc22e5beba0cb71468df1c6b2c184a1777904
Sep 18 14:12:37.711817 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.711788    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7" event={"ID":"5f1fcb61-15a1-493b-af21-6198786c17bd","Type":"ContainerStarted","Data":"2c9ab5dda2023ccdd1d17655b1bcc22e5beba0cb71468df1c6b2c184a1777904"}
Sep 18 14:12:37.956099 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.956040    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"]
Sep 18 14:12:37.959426 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.959406    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:12:37.962877 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.962821    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b6289c\""
Sep 18 14:12:37.973620 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:37.973588    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"]
Sep 18 14:12:38.057619 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.057585    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e729f446-d6f7-4846-a93f-7c5d58148515-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5\" (UID: \"e729f446-d6f7-4846-a93f-7c5d58148515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:12:38.057787 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.057663    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwfh\" (UniqueName: \"kubernetes.io/projected/e729f446-d6f7-4846-a93f-7c5d58148515-kube-api-access-hcwfh\") pod \"payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5\" (UID: \"e729f446-d6f7-4846-a93f-7c5d58148515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:12:38.158780 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.158751    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwfh\" (UniqueName: \"kubernetes.io/projected/e729f446-d6f7-4846-a93f-7c5d58148515-kube-api-access-hcwfh\") pod \"payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5\" (UID: \"e729f446-d6f7-4846-a93f-7c5d58148515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:12:38.158942 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.158842    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e729f446-d6f7-4846-a93f-7c5d58148515-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5\" (UID: \"e729f446-d6f7-4846-a93f-7c5d58148515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:12:38.159517 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.159498    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e729f446-d6f7-4846-a93f-7c5d58148515-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5\" (UID: \"e729f446-d6f7-4846-a93f-7c5d58148515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:12:38.171034 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.171004    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwfh\" (UniqueName: \"kubernetes.io/projected/e729f446-d6f7-4846-a93f-7c5d58148515-kube-api-access-hcwfh\") pod \"payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5\" (UID: \"e729f446-d6f7-4846-a93f-7c5d58148515\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:12:38.269915 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.269830    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:12:38.550443 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.550368    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"]
Sep 18 14:12:38.552650 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:12:38.552619    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode729f446_d6f7_4846_a93f_7c5d58148515.slice/crio-68f714c4b95ad37d99e6c472464463e3dddf5794deaeda3bd7bf61a90cef4acb WatchSource:0}: Error finding container 68f714c4b95ad37d99e6c472464463e3dddf5794deaeda3bd7bf61a90cef4acb: Status 404 returned error can't find the container with id 68f714c4b95ad37d99e6c472464463e3dddf5794deaeda3bd7bf61a90cef4acb
Sep 18 14:12:38.718557 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.718513    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5" event={"ID":"e729f446-d6f7-4846-a93f-7c5d58148515","Type":"ContainerStarted","Data":"83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d"}
Sep 18 14:12:38.718750 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.718570    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5" event={"ID":"e729f446-d6f7-4846-a93f-7c5d58148515","Type":"ContainerStarted","Data":"68f714c4b95ad37d99e6c472464463e3dddf5794deaeda3bd7bf61a90cef4acb"}
Sep 18 14:12:38.718750 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.718599    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:12:38.720249 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.720221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7" event={"ID":"5f1fcb61-15a1-493b-af21-6198786c17bd","Type":"ContainerStarted","Data":"963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32"}
Sep 18 14:12:38.720389 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.720327    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:38.755338 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.755293    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5" podStartSLOduration=1.755278632 podStartE2EDuration="1.755278632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:12:38.753020477 +0000 UTC m=+1700.655476664" watchObservedRunningTime="2026-09-18 14:12:38.755278632 +0000 UTC m=+1700.657734856"
Sep 18 14:12:38.779295 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:38.779250    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7" podStartSLOduration=2.334437775 podStartE2EDuration="2.779232684s" totalImagesPullingTime="444.794909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:12:38.779061048 +0000 UTC m=+1700.681517238" watchObservedRunningTime="2026-09-18 14:12:38.779232684 +0000 UTC m=+1700.681688870"
Sep 18 14:12:44.662574 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:44.662471    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25" podUID="61191bdf-ae6a-4377-b969-9ebd1cbca893" containerName="main" containerID="cri-o://cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237" gracePeriod=45
Sep 18 14:12:44.731265 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:44.731232    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:12:44.924513 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:44.924456    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:12:45.028958 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.028929    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-home\") pod \"61191bdf-ae6a-4377-b969-9ebd1cbca893\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") "
Sep 18 14:12:45.029141 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029009    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxmdm\" (UniqueName: \"kubernetes.io/projected/61191bdf-ae6a-4377-b969-9ebd1cbca893-kube-api-access-kxmdm\") pod \"61191bdf-ae6a-4377-b969-9ebd1cbca893\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") "
Sep 18 14:12:45.029141 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029037    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-tmp-dir\") pod \"61191bdf-ae6a-4377-b969-9ebd1cbca893\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") "
Sep 18 14:12:45.029141 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029092    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-model-cache\") pod \"61191bdf-ae6a-4377-b969-9ebd1cbca893\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") "
Sep 18 14:12:45.029320 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029165    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-dshm\") pod \"61191bdf-ae6a-4377-b969-9ebd1cbca893\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") "
Sep 18 14:12:45.029320 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61191bdf-ae6a-4377-b969-9ebd1cbca893-tls-certs\") pod \"61191bdf-ae6a-4377-b969-9ebd1cbca893\" (UID: \"61191bdf-ae6a-4377-b969-9ebd1cbca893\") "
Sep 18 14:12:45.029320 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029213    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-home" (OuterVolumeSpecName: "home") pod "61191bdf-ae6a-4377-b969-9ebd1cbca893" (UID: "61191bdf-ae6a-4377-b969-9ebd1cbca893"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:45.029475 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029336    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61191bdf-ae6a-4377-b969-9ebd1cbca893" (UID: "61191bdf-ae6a-4377-b969-9ebd1cbca893"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:45.029475 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029380    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-model-cache" (OuterVolumeSpecName: "model-cache") pod "61191bdf-ae6a-4377-b969-9ebd1cbca893" (UID: "61191bdf-ae6a-4377-b969-9ebd1cbca893"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:45.029585 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029528    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-home\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:12:45.029585 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029548    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-tmp-dir\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:12:45.029585 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.029564    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-model-cache\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:12:45.031393 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.031374    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-dshm" (OuterVolumeSpecName: "dshm") pod "61191bdf-ae6a-4377-b969-9ebd1cbca893" (UID: "61191bdf-ae6a-4377-b969-9ebd1cbca893"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:12:45.031456 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.031423    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61191bdf-ae6a-4377-b969-9ebd1cbca893-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61191bdf-ae6a-4377-b969-9ebd1cbca893" (UID: "61191bdf-ae6a-4377-b969-9ebd1cbca893"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:45.031516 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.031501    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61191bdf-ae6a-4377-b969-9ebd1cbca893-kube-api-access-kxmdm" (OuterVolumeSpecName: "kube-api-access-kxmdm") pod "61191bdf-ae6a-4377-b969-9ebd1cbca893" (UID: "61191bdf-ae6a-4377-b969-9ebd1cbca893"). InnerVolumeSpecName "kube-api-access-kxmdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:45.130577 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.130544    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61191bdf-ae6a-4377-b969-9ebd1cbca893-dshm\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:12:45.130577 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.130570    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61191bdf-ae6a-4377-b969-9ebd1cbca893-tls-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:12:45.130577 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.130581    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmdm\" (UniqueName: \"kubernetes.io/projected/61191bdf-ae6a-4377-b969-9ebd1cbca893-kube-api-access-kxmdm\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:12:45.763731 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.763692    2569 generic.go:356] "Generic (PLEG): container finished" podID="61191bdf-ae6a-4377-b969-9ebd1cbca893" containerID="cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237" exitCode=0
Sep 18 14:12:45.764229 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.763740    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"
Sep 18 14:12:45.764229 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.763741    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25" event={"ID":"61191bdf-ae6a-4377-b969-9ebd1cbca893","Type":"ContainerDied","Data":"cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237"}
Sep 18 14:12:45.764229 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.763840    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25" event={"ID":"61191bdf-ae6a-4377-b969-9ebd1cbca893","Type":"ContainerDied","Data":"3837dfcfcf60163e420ddfb75fbc14c1649bdbee26bb972addb3a773b39fee10"}
Sep 18 14:12:45.764229 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.763861    2569 scope.go:117] "RemoveContainer" containerID="cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237"
Sep 18 14:12:45.774330 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.774305    2569 scope.go:117] "RemoveContainer" containerID="cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237"
Sep 18 14:12:45.775417 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:12:45.775384    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237\": container with ID starting with cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237 not found: ID does not exist" containerID="cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237"
Sep 18 14:12:45.775533 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.775437    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237"} err="failed to get container status \"cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237\": rpc error: code = NotFound desc = could not find container \"cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237\": container with ID starting with cc58122a7f4cdbe2dd2d994adc4d5125cb587020762d755154c98b9297783237 not found: ID does not exist"
Sep 18 14:12:45.788098 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.788060    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"]
Sep 18 14:12:45.790158 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:45.790138    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-783872/rate-test-model-783872-kserve-559ddf5779-59r25"]
Sep 18 14:12:46.485860 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:46.485826    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61191bdf-ae6a-4377-b969-9ebd1cbca893" path="/var/lib/kubelet/pods/61191bdf-ae6a-4377-b969-9ebd1cbca893/volumes"
Sep 18 14:12:49.727291 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:49.727261    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:12:50.736234 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:50.736190    2569 patch_prober.go:28] interesting pod/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 14:12:50.736616 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:50.736261    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 14:12:51.795795 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:51.795766    2569 generic.go:356] "Generic (PLEG): container finished" podID="43998e37-a725-46d1-881c-721ac526aff2" containerID="777c21aa3b7caf7325db41e1b58496154f4a4a45f3c04f0bbfe74bfc5bb55297" exitCode=0
Sep 18 14:12:51.796235 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:51.795834    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" event={"ID":"43998e37-a725-46d1-881c-721ac526aff2","Type":"ContainerDied","Data":"777c21aa3b7caf7325db41e1b58496154f4a4a45f3c04f0bbfe74bfc5bb55297"}
Sep 18 14:12:51.796299 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:51.796284    2569 scope.go:117] "RemoveContainer" containerID="777c21aa3b7caf7325db41e1b58496154f4a4a45f3c04f0bbfe74bfc5bb55297"
Sep 18 14:12:51.797360 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:51.797335    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:12:51.797418 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:51.797400    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:12:51.797460 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:51.797448    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 14:12:52.802235 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:52.802193    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" event={"ID":"43998e37-a725-46d1-881c-721ac526aff2","Type":"ContainerStarted","Data":"e9263553957a37dfb0942c3e9aa682b35df6e7af89385aa66b021e73cc61370d"}
Sep 18 14:12:53.752368 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:53.752326    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:53.752582 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:53.752458    2569 patch_prober.go:28] interesting pod/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 14:12:53.752582 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:53.752501    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 14:12:54.751899 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:54.751867    2569 patch_prober.go:28] interesting pod/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 14:12:54.752517 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:54.752467    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 14:12:55.752354 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:55.752307    2569 patch_prober.go:28] interesting pod/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 14:12:55.752863 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:55.752379    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 14:12:56.752305 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:56.752270    2569 patch_prober.go:28] interesting pod/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 14:12:56.752604 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:56.752575    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 14:12:57.752851 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:57.752813    2569 patch_prober.go:28] interesting pod/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 14:12:57.753489 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:57.753455    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 14:12:58.818772 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:58.818740    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:58.819216 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:58.819148    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:12:58.819977 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:12:58.819957    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:13:06.009267 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.009231    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"]
Sep 18 14:13:06.009789 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.009598    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61191bdf-ae6a-4377-b969-9ebd1cbca893" containerName="main"
Sep 18 14:13:06.009789 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.009610    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="61191bdf-ae6a-4377-b969-9ebd1cbca893" containerName="main"
Sep 18 14:13:06.009789 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.009670    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="61191bdf-ae6a-4377-b969-9ebd1cbca893" containerName="main"
Sep 18 14:13:06.012318 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.012293    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.015376 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.015353    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9d6301-serving-cert\""
Sep 18 14:13:06.027226 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.027202    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"]
Sep 18 14:13:06.059980 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.059951    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed52a62-3805-443f-bb1f-144c08b30c67-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.060163 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.059992    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.060163 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.060031    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.060163 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.060104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jncf6\" (UniqueName: \"kubernetes.io/projected/1ed52a62-3805-443f-bb1f-144c08b30c67-kube-api-access-jncf6\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.161480 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.161442    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jncf6\" (UniqueName: \"kubernetes.io/projected/1ed52a62-3805-443f-bb1f-144c08b30c67-kube-api-access-jncf6\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.161675 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.161510    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed52a62-3805-443f-bb1f-144c08b30c67-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.161675 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.161543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.161675 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.161586    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.161859 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:13:06.161690    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-serving-cert: secret "maas-api-e2e-ipp-b-9d6301-serving-cert" not found
Sep 18 14:13:06.161859 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:13:06.161782    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-maas-api-tls podName:1ed52a62-3805-443f-bb1f-144c08b30c67 nodeName:}" failed. No retries permitted until 2026-09-18 14:13:06.661760227 +0000 UTC m=+1728.564216403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-maas-api-tls") pod "maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh" (UID: "1ed52a62-3805-443f-bb1f-144c08b30c67") : secret "maas-api-e2e-ipp-b-9d6301-serving-cert" not found
Sep 18 14:13:06.162371 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.162349    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed52a62-3805-443f-bb1f-144c08b30c67-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.164376 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.164357    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.170822 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.170785    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jncf6\" (UniqueName: \"kubernetes.io/projected/1ed52a62-3805-443f-bb1f-144c08b30c67-kube-api-access-jncf6\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.667007 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.666972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.669665 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.669634    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.744103 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.744049    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"]
Sep 18 14:13:06.746509 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.746493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:13:06.749828 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.749807    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9d6301\""
Sep 18 14:13:06.749927 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.749884    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9d6301-dockercfg-jg7p2\""
Sep 18 14:13:06.759690 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.759668    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"]
Sep 18 14:13:06.869111 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.869051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9476\" (UniqueName: \"kubernetes.io/projected/43c2096e-f609-4ec1-9ae6-c4e607d35d67-kube-api-access-l9476\") pod \"payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w\" (UID: \"43c2096e-f609-4ec1-9ae6-c4e607d35d67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:13:06.869288 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.869155    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c2096e-f609-4ec1-9ae6-c4e607d35d67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w\" (UID: \"43c2096e-f609-4ec1-9ae6-c4e607d35d67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:13:06.922701 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.922627    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:06.970109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.970053    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9476\" (UniqueName: \"kubernetes.io/projected/43c2096e-f609-4ec1-9ae6-c4e607d35d67-kube-api-access-l9476\") pod \"payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w\" (UID: \"43c2096e-f609-4ec1-9ae6-c4e607d35d67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:13:06.970282 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.970136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c2096e-f609-4ec1-9ae6-c4e607d35d67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w\" (UID: \"43c2096e-f609-4ec1-9ae6-c4e607d35d67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:13:06.970893 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.970868    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c2096e-f609-4ec1-9ae6-c4e607d35d67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w\" (UID: \"43c2096e-f609-4ec1-9ae6-c4e607d35d67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:13:06.980007 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:06.979976    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9476\" (UniqueName: \"kubernetes.io/projected/43c2096e-f609-4ec1-9ae6-c4e607d35d67-kube-api-access-l9476\") pod \"payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w\" (UID: \"43c2096e-f609-4ec1-9ae6-c4e607d35d67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:13:07.056909 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.056885    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:13:07.199411 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.199385    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"]
Sep 18 14:13:07.201886 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:13:07.201859    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed52a62_3805_443f_bb1f_144c08b30c67.slice/crio-f4ab51ea0c16a00460054b9a49da1d426e39da0a87fb0b9f6c8c91113ace8a16 WatchSource:0}: Error finding container f4ab51ea0c16a00460054b9a49da1d426e39da0a87fb0b9f6c8c91113ace8a16: Status 404 returned error can't find the container with id f4ab51ea0c16a00460054b9a49da1d426e39da0a87fb0b9f6c8c91113ace8a16
Sep 18 14:13:07.329434 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.329407    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"]
Sep 18 14:13:07.331700 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:13:07.331670    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c2096e_f609_4ec1_9ae6_c4e607d35d67.slice/crio-1c70ef2d36b1174a4d4dfe52fc691456b5cd3c1e2b00a57023133b1c643abcb2 WatchSource:0}: Error finding container 1c70ef2d36b1174a4d4dfe52fc691456b5cd3c1e2b00a57023133b1c643abcb2: Status 404 returned error can't find the container with id 1c70ef2d36b1174a4d4dfe52fc691456b5cd3c1e2b00a57023133b1c643abcb2
Sep 18 14:13:07.880736 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.880645    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh" event={"ID":"1ed52a62-3805-443f-bb1f-144c08b30c67","Type":"ContainerStarted","Data":"ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405"}
Sep 18 14:13:07.880736 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.880704    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:07.880736 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.880725    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh" event={"ID":"1ed52a62-3805-443f-bb1f-144c08b30c67","Type":"ContainerStarted","Data":"f4ab51ea0c16a00460054b9a49da1d426e39da0a87fb0b9f6c8c91113ace8a16"}
Sep 18 14:13:07.882456 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.882424    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w" event={"ID":"43c2096e-f609-4ec1-9ae6-c4e607d35d67","Type":"ContainerStarted","Data":"d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff"}
Sep 18 14:13:07.882605 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.882473    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:13:07.882605 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.882496    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w" event={"ID":"43c2096e-f609-4ec1-9ae6-c4e607d35d67","Type":"ContainerStarted","Data":"1c70ef2d36b1174a4d4dfe52fc691456b5cd3c1e2b00a57023133b1c643abcb2"}
Sep 18 14:13:07.899343 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.899301    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh" podStartSLOduration=2.551795042 podStartE2EDuration="2.899288134s" totalImagesPullingTime="347.493092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:13:07.898900893 +0000 UTC m=+1729.801357097" watchObservedRunningTime="2026-09-18 14:13:07.899288134 +0000 UTC m=+1729.801744320"
Sep 18 14:13:07.917408 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:07.917356    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w" podStartSLOduration=1.917336813 podStartE2EDuration="1.917336813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:13:07.916894257 +0000 UTC m=+1729.819350444" watchObservedRunningTime="2026-09-18 14:13:07.917336813 +0000 UTC m=+1729.819793003"
Sep 18 14:13:13.893509 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:13.893472    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:13:18.890659 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:18.890630    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:13:33.714021 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:13:33.713988    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w_43c2096e-f609-4ec1-9ae6-c4e607d35d67/payload-processing/0.log"
Sep 18 14:14:15.043037 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:15.042969    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w_43c2096e-f609-4ec1-9ae6-c4e607d35d67/payload-processing/0.log"
Sep 18 14:14:15.161417 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:15.161387    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w_43c2096e-f609-4ec1-9ae6-c4e607d35d67/payload-processing/0.log"
Sep 18 14:14:18.615334 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:18.615305    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 14:14:18.615876 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:18.615859    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 14:14:23.015764 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.015721    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"]
Sep 18 14:14:23.018364 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.018348    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:23.021309 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.021290    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-cafbd6\""
Sep 18 14:14:23.021481 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.021464    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-cafbd6-dockercfg-txr6q\""
Sep 18 14:14:23.030319 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.030296    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"]
Sep 18 14:14:23.091899 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.091861    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee5a6e5-40d1-4d53-96a9-a190b3bed501-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77\" (UID: \"cee5a6e5-40d1-4d53-96a9-a190b3bed501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:23.092100 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.091914    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxm6\" (UniqueName: \"kubernetes.io/projected/cee5a6e5-40d1-4d53-96a9-a190b3bed501-kube-api-access-ddxm6\") pod \"payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77\" (UID: \"cee5a6e5-40d1-4d53-96a9-a190b3bed501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:23.192445 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.192402    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxm6\" (UniqueName: \"kubernetes.io/projected/cee5a6e5-40d1-4d53-96a9-a190b3bed501-kube-api-access-ddxm6\") pod \"payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77\" (UID: \"cee5a6e5-40d1-4d53-96a9-a190b3bed501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:23.192659 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.192543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee5a6e5-40d1-4d53-96a9-a190b3bed501-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77\" (UID: \"cee5a6e5-40d1-4d53-96a9-a190b3bed501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:23.193269 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.193244    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee5a6e5-40d1-4d53-96a9-a190b3bed501-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77\" (UID: \"cee5a6e5-40d1-4d53-96a9-a190b3bed501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:23.201613 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.201588    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxm6\" (UniqueName: \"kubernetes.io/projected/cee5a6e5-40d1-4d53-96a9-a190b3bed501-kube-api-access-ddxm6\") pod \"payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77\" (UID: \"cee5a6e5-40d1-4d53-96a9-a190b3bed501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:23.329906 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.329805    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:23.605592 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:23.605567    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"]
Sep 18 14:14:23.607873 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:14:23.607847    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee5a6e5_40d1_4d53_96a9_a190b3bed501.slice/crio-1dd9db683fef7493eed4fea45ccb4e1255dc6e32efc55bf884dd6cf0c1da35ca WatchSource:0}: Error finding container 1dd9db683fef7493eed4fea45ccb4e1255dc6e32efc55bf884dd6cf0c1da35ca: Status 404 returned error can't find the container with id 1dd9db683fef7493eed4fea45ccb4e1255dc6e32efc55bf884dd6cf0c1da35ca
Sep 18 14:14:24.268184 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:24.268142    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77" event={"ID":"cee5a6e5-40d1-4d53-96a9-a190b3bed501","Type":"ContainerStarted","Data":"98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1"}
Sep 18 14:14:24.268606 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:24.268210    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77" event={"ID":"cee5a6e5-40d1-4d53-96a9-a190b3bed501","Type":"ContainerStarted","Data":"1dd9db683fef7493eed4fea45ccb4e1255dc6e32efc55bf884dd6cf0c1da35ca"}
Sep 18 14:14:24.268606 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:24.268323    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:24.288235 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:24.288192    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77" podStartSLOduration=2.2881765339999998 podStartE2EDuration="2.288176534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:14:24.287354758 +0000 UTC m=+1806.189810944" watchObservedRunningTime="2026-09-18 14:14:24.288176534 +0000 UTC m=+1806.190632720"
Sep 18 14:14:35.275261 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:35.275233    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:51.319993 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:51.319959    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"]
Sep 18 14:14:51.320612 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:51.320237    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77" podUID="cee5a6e5-40d1-4d53-96a9-a190b3bed501" containerName="payload-pre-processing" containerID="cri-o://98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1" gracePeriod=30
Sep 18 14:14:51.581837 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:51.581763    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:51.645011 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:51.644977    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee5a6e5-40d1-4d53-96a9-a190b3bed501-plugins-config-volume\") pod \"cee5a6e5-40d1-4d53-96a9-a190b3bed501\" (UID: \"cee5a6e5-40d1-4d53-96a9-a190b3bed501\") "
Sep 18 14:14:51.645011 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:51.645016    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddxm6\" (UniqueName: \"kubernetes.io/projected/cee5a6e5-40d1-4d53-96a9-a190b3bed501-kube-api-access-ddxm6\") pod \"cee5a6e5-40d1-4d53-96a9-a190b3bed501\" (UID: \"cee5a6e5-40d1-4d53-96a9-a190b3bed501\") "
Sep 18 14:14:51.645477 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:51.645449    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee5a6e5-40d1-4d53-96a9-a190b3bed501-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cee5a6e5-40d1-4d53-96a9-a190b3bed501" (UID: "cee5a6e5-40d1-4d53-96a9-a190b3bed501"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:14:51.647435 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:51.647413    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee5a6e5-40d1-4d53-96a9-a190b3bed501-kube-api-access-ddxm6" (OuterVolumeSpecName: "kube-api-access-ddxm6") pod "cee5a6e5-40d1-4d53-96a9-a190b3bed501" (UID: "cee5a6e5-40d1-4d53-96a9-a190b3bed501"). InnerVolumeSpecName "kube-api-access-ddxm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:14:51.745941 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:51.745914    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee5a6e5-40d1-4d53-96a9-a190b3bed501-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:14:51.745941 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:51.745938    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddxm6\" (UniqueName: \"kubernetes.io/projected/cee5a6e5-40d1-4d53-96a9-a190b3bed501-kube-api-access-ddxm6\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:14:52.407977 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:52.407942    2569 generic.go:356] "Generic (PLEG): container finished" podID="cee5a6e5-40d1-4d53-96a9-a190b3bed501" containerID="98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1" exitCode=0
Sep 18 14:14:52.407977 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:52.407980    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77" event={"ID":"cee5a6e5-40d1-4d53-96a9-a190b3bed501","Type":"ContainerDied","Data":"98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1"}
Sep 18 14:14:52.408503 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:52.407989    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"
Sep 18 14:14:52.408503 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:52.408023    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77" event={"ID":"cee5a6e5-40d1-4d53-96a9-a190b3bed501","Type":"ContainerDied","Data":"1dd9db683fef7493eed4fea45ccb4e1255dc6e32efc55bf884dd6cf0c1da35ca"}
Sep 18 14:14:52.408503 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:52.408045    2569 scope.go:117] "RemoveContainer" containerID="98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1"
Sep 18 14:14:52.420091 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:52.419587    2569 scope.go:117] "RemoveContainer" containerID="98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1"
Sep 18 14:14:52.420091 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:14:52.419972    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1\": container with ID starting with 98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1 not found: ID does not exist" containerID="98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1"
Sep 18 14:14:52.420091 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:52.420007    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1"} err="failed to get container status \"98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1\": rpc error: code = NotFound desc = could not find container \"98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1\": container with ID starting with 98cbf4b52f6ff6a13cbd5bc095df2730f9ed3970b43d6bda6ecc6f9bfba219c1 not found: ID does not exist"
Sep 18 14:14:52.433229 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:52.433202    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"]
Sep 18 14:14:52.436479 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:52.436456    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cafbd6-69cc64dc6b-dsl77"]
Sep 18 14:14:52.486166 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:14:52.486137    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cee5a6e5-40d1-4d53-96a9-a190b3bed501" path="/var/lib/kubelet/pods/cee5a6e5-40d1-4d53-96a9-a190b3bed501/volumes"
Sep 18 14:15:17.834814 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:17.834779    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"]
Sep 18 14:15:17.835668 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:17.835630    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7" podUID="5f1fcb61-15a1-493b-af21-6198786c17bd" containerName="maas-api" containerID="cri-o://963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32" gracePeriod=30
Sep 18 14:15:17.894836 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:17.894795    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"]
Sep 18 14:15:17.895123 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:17.895094    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5" podUID="e729f446-d6f7-4846-a93f-7c5d58148515" containerName="payload-pre-processing" containerID="cri-o://83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d" gracePeriod=30
Sep 18 14:15:18.147615 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.147462    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:15:18.179768 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.179735    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7hth\" (UniqueName: \"kubernetes.io/projected/5f1fcb61-15a1-493b-af21-6198786c17bd-kube-api-access-w7hth\") pod \"5f1fcb61-15a1-493b-af21-6198786c17bd\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") "
Sep 18 14:15:18.179913 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.179810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1fcb61-15a1-493b-af21-6198786c17bd-openshift-service-ca\") pod \"5f1fcb61-15a1-493b-af21-6198786c17bd\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") "
Sep 18 14:15:18.179913 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.179857    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-maas-api-tls\") pod \"5f1fcb61-15a1-493b-af21-6198786c17bd\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") "
Sep 18 14:15:18.179913 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.179894    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-metrics-certs\") pod \"5f1fcb61-15a1-493b-af21-6198786c17bd\" (UID: \"5f1fcb61-15a1-493b-af21-6198786c17bd\") "
Sep 18 14:15:18.180380 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.180259    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1fcb61-15a1-493b-af21-6198786c17bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f1fcb61-15a1-493b-af21-6198786c17bd" (UID: "5f1fcb61-15a1-493b-af21-6198786c17bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:15:18.183335 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.183308    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5f1fcb61-15a1-493b-af21-6198786c17bd" (UID: "5f1fcb61-15a1-493b-af21-6198786c17bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:15:18.183456 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.183392    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1fcb61-15a1-493b-af21-6198786c17bd-kube-api-access-w7hth" (OuterVolumeSpecName: "kube-api-access-w7hth") pod "5f1fcb61-15a1-493b-af21-6198786c17bd" (UID: "5f1fcb61-15a1-493b-af21-6198786c17bd"). InnerVolumeSpecName "kube-api-access-w7hth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:15:18.185748 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.185722    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f1fcb61-15a1-493b-af21-6198786c17bd" (UID: "5f1fcb61-15a1-493b-af21-6198786c17bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:15:18.189482 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.189463    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:15:18.280963 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.280930    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e729f446-d6f7-4846-a93f-7c5d58148515-plugins-config-volume\") pod \"e729f446-d6f7-4846-a93f-7c5d58148515\" (UID: \"e729f446-d6f7-4846-a93f-7c5d58148515\") "
Sep 18 14:15:18.281246 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.280981    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcwfh\" (UniqueName: \"kubernetes.io/projected/e729f446-d6f7-4846-a93f-7c5d58148515-kube-api-access-hcwfh\") pod \"e729f446-d6f7-4846-a93f-7c5d58148515\" (UID: \"e729f446-d6f7-4846-a93f-7c5d58148515\") "
Sep 18 14:15:18.281356 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.281331    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7hth\" (UniqueName: \"kubernetes.io/projected/5f1fcb61-15a1-493b-af21-6198786c17bd-kube-api-access-w7hth\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:15:18.281422 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.281357    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1fcb61-15a1-493b-af21-6198786c17bd-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:15:18.281422 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.281354    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e729f446-d6f7-4846-a93f-7c5d58148515-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e729f446-d6f7-4846-a93f-7c5d58148515" (UID: "e729f446-d6f7-4846-a93f-7c5d58148515"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:15:18.281422 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.281373    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:15:18.281422 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.281387    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1fcb61-15a1-493b-af21-6198786c17bd-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:15:18.284052 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.284010    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e729f446-d6f7-4846-a93f-7c5d58148515-kube-api-access-hcwfh" (OuterVolumeSpecName: "kube-api-access-hcwfh") pod "e729f446-d6f7-4846-a93f-7c5d58148515" (UID: "e729f446-d6f7-4846-a93f-7c5d58148515"). InnerVolumeSpecName "kube-api-access-hcwfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:15:18.382730 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.382656    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e729f446-d6f7-4846-a93f-7c5d58148515-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:15:18.382730 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.382684    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcwfh\" (UniqueName: \"kubernetes.io/projected/e729f446-d6f7-4846-a93f-7c5d58148515-kube-api-access-hcwfh\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:15:18.537756 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.537721    2569 generic.go:356] "Generic (PLEG): container finished" podID="5f1fcb61-15a1-493b-af21-6198786c17bd" containerID="963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32" exitCode=0
Sep 18 14:15:18.537883 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.537780    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"
Sep 18 14:15:18.537883 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.537801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7" event={"ID":"5f1fcb61-15a1-493b-af21-6198786c17bd","Type":"ContainerDied","Data":"963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32"}
Sep 18 14:15:18.537883 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.537853    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7" event={"ID":"5f1fcb61-15a1-493b-af21-6198786c17bd","Type":"ContainerDied","Data":"2c9ab5dda2023ccdd1d17655b1bcc22e5beba0cb71468df1c6b2c184a1777904"}
Sep 18 14:15:18.537883 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.537879    2569 scope.go:117] "RemoveContainer" containerID="963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32"
Sep 18 14:15:18.539253 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.539227    2569 generic.go:356] "Generic (PLEG): container finished" podID="e729f446-d6f7-4846-a93f-7c5d58148515" containerID="83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d" exitCode=0
Sep 18 14:15:18.539389 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.539263    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5" event={"ID":"e729f446-d6f7-4846-a93f-7c5d58148515","Type":"ContainerDied","Data":"83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d"}
Sep 18 14:15:18.539389 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.539292    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5" event={"ID":"e729f446-d6f7-4846-a93f-7c5d58148515","Type":"ContainerDied","Data":"68f714c4b95ad37d99e6c472464463e3dddf5794deaeda3bd7bf61a90cef4acb"}
Sep 18 14:15:18.539389 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.539356    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"
Sep 18 14:15:18.547025 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.547008    2569 scope.go:117] "RemoveContainer" containerID="963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32"
Sep 18 14:15:18.547337 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:15:18.547306    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32\": container with ID starting with 963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32 not found: ID does not exist" containerID="963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32"
Sep 18 14:15:18.547428 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.547348    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32"} err="failed to get container status \"963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32\": rpc error: code = NotFound desc = could not find container \"963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32\": container with ID starting with 963c2d1151a1b08c8e83f0e511656264c8df81fd2c89d63502e384fc9782fc32 not found: ID does not exist"
Sep 18 14:15:18.547428 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.547370    2569 scope.go:117] "RemoveContainer" containerID="83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d"
Sep 18 14:15:18.555158 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.555140    2569 scope.go:117] "RemoveContainer" containerID="83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d"
Sep 18 14:15:18.555425 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:15:18.555405    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d\": container with ID starting with 83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d not found: ID does not exist" containerID="83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d"
Sep 18 14:15:18.555490 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.555437    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d"} err="failed to get container status \"83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d\": rpc error: code = NotFound desc = could not find container \"83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d\": container with ID starting with 83af51b415734f1f7eaf08de17563b2e81b6aa1e278ab0e7ef30829d2576239d not found: ID does not exist"
Sep 18 14:15:18.559828 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.559800    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"]
Sep 18 14:15:18.566497 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.566348    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b6289c-6fbf9fdf75-9n7d7"]
Sep 18 14:15:18.577627 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.577598    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"]
Sep 18 14:15:18.586699 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:18.586672    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b6289c-5fcfd4c9f8-vjlm5"]
Sep 18 14:15:20.486359 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:20.486328    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f1fcb61-15a1-493b-af21-6198786c17bd" path="/var/lib/kubelet/pods/5f1fcb61-15a1-493b-af21-6198786c17bd/volumes"
Sep 18 14:15:20.486722 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:15:20.486691    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e729f446-d6f7-4846-a93f-7c5d58148515" path="/var/lib/kubelet/pods/e729f446-d6f7-4846-a93f-7c5d58148515/volumes"
Sep 18 14:16:27.692395 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:27.692351    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"]
Sep 18 14:16:27.692942 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:27.692613    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy" containerID="cri-o://e9263553957a37dfb0942c3e9aa682b35df6e7af89385aa66b021e73cc61370d" gracePeriod=30
Sep 18 14:16:28.821248 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:28.821210    2569 patch_prober.go:28] interesting pod/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 14:16:28.821650 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:28.821281    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:16:32.897429 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:32.897389    2569 generic.go:356] "Generic (PLEG): container finished" podID="43998e37-a725-46d1-881c-721ac526aff2" containerID="e9263553957a37dfb0942c3e9aa682b35df6e7af89385aa66b021e73cc61370d" exitCode=0
Sep 18 14:16:32.897796 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:32.897456    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" event={"ID":"43998e37-a725-46d1-881c-721ac526aff2","Type":"ContainerDied","Data":"e9263553957a37dfb0942c3e9aa682b35df6e7af89385aa66b021e73cc61370d"}
Sep 18 14:16:32.897796 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:32.897515    2569 scope.go:117] "RemoveContainer" containerID="777c21aa3b7caf7325db41e1b58496154f4a4a45f3c04f0bbfe74bfc5bb55297"
Sep 18 14:16:32.949486 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:32.949461    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:16:33.040310 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040229    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-certs\") pod \"43998e37-a725-46d1-881c-721ac526aff2\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") "
Sep 18 14:16:33.040310 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040282    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-istio-token\") pod \"43998e37-a725-46d1-881c-721ac526aff2\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") "
Sep 18 14:16:33.040310 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040310    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-credential-socket\") pod \"43998e37-a725-46d1-881c-721ac526aff2\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") "
Sep 18 14:16:33.040578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040339    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-data\") pod \"43998e37-a725-46d1-881c-721ac526aff2\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") "
Sep 18 14:16:33.040578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040400    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-socket\") pod \"43998e37-a725-46d1-881c-721ac526aff2\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") "
Sep 18 14:16:33.040578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040427    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-envoy\") pod \"43998e37-a725-46d1-881c-721ac526aff2\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") "
Sep 18 14:16:33.040578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040455    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43998e37-a725-46d1-881c-721ac526aff2-istio-podinfo\") pod \"43998e37-a725-46d1-881c-721ac526aff2\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") "
Sep 18 14:16:33.040578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040489    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43998e37-a725-46d1-881c-721ac526aff2-istiod-ca-cert\") pod \"43998e37-a725-46d1-881c-721ac526aff2\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") "
Sep 18 14:16:33.040578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040522    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9s55\" (UniqueName: \"kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-kube-api-access-t9s55\") pod \"43998e37-a725-46d1-881c-721ac526aff2\" (UID: \"43998e37-a725-46d1-881c-721ac526aff2\") "
Sep 18 14:16:33.040578 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040530    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "43998e37-a725-46d1-881c-721ac526aff2" (UID: "43998e37-a725-46d1-881c-721ac526aff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:33.040951 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040745    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "43998e37-a725-46d1-881c-721ac526aff2" (UID: "43998e37-a725-46d1-881c-721ac526aff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:33.040951 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040801    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.040951 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040821    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43998e37-a725-46d1-881c-721ac526aff2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "43998e37-a725-46d1-881c-721ac526aff2" (UID: "43998e37-a725-46d1-881c-721ac526aff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:16:33.041141 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.040996    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-data" (OuterVolumeSpecName: "istio-data") pod "43998e37-a725-46d1-881c-721ac526aff2" (UID: "43998e37-a725-46d1-881c-721ac526aff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:33.041141 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.041016    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "43998e37-a725-46d1-881c-721ac526aff2" (UID: "43998e37-a725-46d1-881c-721ac526aff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:33.042686 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.042665    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-istio-token" (OuterVolumeSpecName: "istio-token") pod "43998e37-a725-46d1-881c-721ac526aff2" (UID: "43998e37-a725-46d1-881c-721ac526aff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:33.042814 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.042767    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "43998e37-a725-46d1-881c-721ac526aff2" (UID: "43998e37-a725-46d1-881c-721ac526aff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:33.042814 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.042778    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-kube-api-access-t9s55" (OuterVolumeSpecName: "kube-api-access-t9s55") pod "43998e37-a725-46d1-881c-721ac526aff2" (UID: "43998e37-a725-46d1-881c-721ac526aff2"). InnerVolumeSpecName "kube-api-access-t9s55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:33.042900 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.042856    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43998e37-a725-46d1-881c-721ac526aff2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "43998e37-a725-46d1-881c-721ac526aff2" (UID: "43998e37-a725-46d1-881c-721ac526aff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:16:33.142176 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.142138    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-istio-token\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.142176 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.142173    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-credential-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.142176 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.142183    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-data\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.142176 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.142191    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-workload-socket\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.142431 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.142199    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43998e37-a725-46d1-881c-721ac526aff2-istio-envoy\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.142431 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.142207    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43998e37-a725-46d1-881c-721ac526aff2-istio-podinfo\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.142431 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.142214    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43998e37-a725-46d1-881c-721ac526aff2-istiod-ca-cert\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.142431 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.142222    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9s55\" (UniqueName: \"kubernetes.io/projected/43998e37-a725-46d1-881c-721ac526aff2-kube-api-access-t9s55\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:33.902824 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.902787    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz" event={"ID":"43998e37-a725-46d1-881c-721ac526aff2","Type":"ContainerDied","Data":"dab8ad59b5e6a6f0e50a90fac4c3688aedbd50d1a6069ebf16b44109dca63fc6"}
Sep 18 14:16:33.903295 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.902849    2569 scope.go:117] "RemoveContainer" containerID="e9263553957a37dfb0942c3e9aa682b35df6e7af89385aa66b021e73cc61370d"
Sep 18 14:16:33.903295 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.902802    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"
Sep 18 14:16:33.926727 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.926704    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"]
Sep 18 14:16:33.931220 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:33.931200    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b6289c-openshift-default-57dc857d54-txqzz"]
Sep 18 14:16:34.486331 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:34.486300    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43998e37-a725-46d1-881c-721ac526aff2" path="/var/lib/kubelet/pods/43998e37-a725-46d1-881c-721ac526aff2/volumes"
Sep 18 14:16:38.494351 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.494318    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"]
Sep 18 14:16:38.494858 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.494563    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh" podUID="1ed52a62-3805-443f-bb1f-144c08b30c67" containerName="maas-api" containerID="cri-o://ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405" gracePeriod=30
Sep 18 14:16:38.558855 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.558824    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"]
Sep 18 14:16:38.754516 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.754458    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:16:38.892080 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.892039    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-metrics-certs\") pod \"1ed52a62-3805-443f-bb1f-144c08b30c67\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") "
Sep 18 14:16:38.892239 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.892108    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-maas-api-tls\") pod \"1ed52a62-3805-443f-bb1f-144c08b30c67\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") "
Sep 18 14:16:38.892239 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.892158    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed52a62-3805-443f-bb1f-144c08b30c67-openshift-service-ca\") pod \"1ed52a62-3805-443f-bb1f-144c08b30c67\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") "
Sep 18 14:16:38.892239 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.892193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jncf6\" (UniqueName: \"kubernetes.io/projected/1ed52a62-3805-443f-bb1f-144c08b30c67-kube-api-access-jncf6\") pod \"1ed52a62-3805-443f-bb1f-144c08b30c67\" (UID: \"1ed52a62-3805-443f-bb1f-144c08b30c67\") "
Sep 18 14:16:38.892525 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.892500    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed52a62-3805-443f-bb1f-144c08b30c67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ed52a62-3805-443f-bb1f-144c08b30c67" (UID: "1ed52a62-3805-443f-bb1f-144c08b30c67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:16:38.894375 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.894348    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1ed52a62-3805-443f-bb1f-144c08b30c67" (UID: "1ed52a62-3805-443f-bb1f-144c08b30c67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:16:38.894495 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.894405    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ed52a62-3805-443f-bb1f-144c08b30c67" (UID: "1ed52a62-3805-443f-bb1f-144c08b30c67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:16:38.894495 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.894430    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed52a62-3805-443f-bb1f-144c08b30c67-kube-api-access-jncf6" (OuterVolumeSpecName: "kube-api-access-jncf6") pod "1ed52a62-3805-443f-bb1f-144c08b30c67" (UID: "1ed52a62-3805-443f-bb1f-144c08b30c67"). InnerVolumeSpecName "kube-api-access-jncf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:38.926916 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.926884    2569 generic.go:356] "Generic (PLEG): container finished" podID="1ed52a62-3805-443f-bb1f-144c08b30c67" containerID="ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405" exitCode=0
Sep 18 14:16:38.927032 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.926935    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"
Sep 18 14:16:38.927032 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.926970    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh" event={"ID":"1ed52a62-3805-443f-bb1f-144c08b30c67","Type":"ContainerDied","Data":"ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405"}
Sep 18 14:16:38.927032 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.927016    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh" event={"ID":"1ed52a62-3805-443f-bb1f-144c08b30c67","Type":"ContainerDied","Data":"f4ab51ea0c16a00460054b9a49da1d426e39da0a87fb0b9f6c8c91113ace8a16"}
Sep 18 14:16:38.927163 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.927040    2569 scope.go:117] "RemoveContainer" containerID="ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405"
Sep 18 14:16:38.927341 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.927308    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w" podUID="43c2096e-f609-4ec1-9ae6-c4e607d35d67" containerName="payload-processing" containerID="cri-o://d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff" gracePeriod=30
Sep 18 14:16:38.937431 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.937107    2569 scope.go:117] "RemoveContainer" containerID="ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405"
Sep 18 14:16:38.937588 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:16:38.937562    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405\": container with ID starting with ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405 not found: ID does not exist" containerID="ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405"
Sep 18 14:16:38.937660 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.937592    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405"} err="failed to get container status \"ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405\": rpc error: code = NotFound desc = could not find container \"ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405\": container with ID starting with ac39cbdc95daaa414d44fb70db2745eb2e227f2ba63d91b69b1878604cc8d405 not found: ID does not exist"
Sep 18 14:16:38.964032 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.964001    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"]
Sep 18 14:16:38.967817 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.967796    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d6301-575b69b78c-vqxnh"]
Sep 18 14:16:38.993304 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.993284    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:38.993304 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.993304    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ed52a62-3805-443f-bb1f-144c08b30c67-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:38.993457 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.993313    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed52a62-3805-443f-bb1f-144c08b30c67-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:38.993457 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:38.993322    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jncf6\" (UniqueName: \"kubernetes.io/projected/1ed52a62-3805-443f-bb1f-144c08b30c67-kube-api-access-jncf6\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:39.169947 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.169926    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:16:39.296285 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.296210    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c2096e-f609-4ec1-9ae6-c4e607d35d67-plugins-config-volume\") pod \"43c2096e-f609-4ec1-9ae6-c4e607d35d67\" (UID: \"43c2096e-f609-4ec1-9ae6-c4e607d35d67\") "
Sep 18 14:16:39.296285 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.296279    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9476\" (UniqueName: \"kubernetes.io/projected/43c2096e-f609-4ec1-9ae6-c4e607d35d67-kube-api-access-l9476\") pod \"43c2096e-f609-4ec1-9ae6-c4e607d35d67\" (UID: \"43c2096e-f609-4ec1-9ae6-c4e607d35d67\") "
Sep 18 14:16:39.296598 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.296574    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c2096e-f609-4ec1-9ae6-c4e607d35d67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43c2096e-f609-4ec1-9ae6-c4e607d35d67" (UID: "43c2096e-f609-4ec1-9ae6-c4e607d35d67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:16:39.298429 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.298411    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c2096e-f609-4ec1-9ae6-c4e607d35d67-kube-api-access-l9476" (OuterVolumeSpecName: "kube-api-access-l9476") pod "43c2096e-f609-4ec1-9ae6-c4e607d35d67" (UID: "43c2096e-f609-4ec1-9ae6-c4e607d35d67"). InnerVolumeSpecName "kube-api-access-l9476". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:39.397397 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.397368    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43c2096e-f609-4ec1-9ae6-c4e607d35d67-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:39.397397 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.397391    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9476\" (UniqueName: \"kubernetes.io/projected/43c2096e-f609-4ec1-9ae6-c4e607d35d67-kube-api-access-l9476\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:39.934174 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.934143    2569 generic.go:356] "Generic (PLEG): container finished" podID="43c2096e-f609-4ec1-9ae6-c4e607d35d67" containerID="d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff" exitCode=0
Sep 18 14:16:39.934587 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.934182    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"
Sep 18 14:16:39.934587 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.934225    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w" event={"ID":"43c2096e-f609-4ec1-9ae6-c4e607d35d67","Type":"ContainerDied","Data":"d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff"}
Sep 18 14:16:39.934587 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.934272    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w" event={"ID":"43c2096e-f609-4ec1-9ae6-c4e607d35d67","Type":"ContainerDied","Data":"1c70ef2d36b1174a4d4dfe52fc691456b5cd3c1e2b00a57023133b1c643abcb2"}
Sep 18 14:16:39.934587 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.934293    2569 scope.go:117] "RemoveContainer" containerID="d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff"
Sep 18 14:16:39.943754 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.943736    2569 scope.go:117] "RemoveContainer" containerID="d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff"
Sep 18 14:16:39.944006 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:16:39.943986    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff\": container with ID starting with d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff not found: ID does not exist" containerID="d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff"
Sep 18 14:16:39.944054 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.944014    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff"} err="failed to get container status \"d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff\": rpc error: code = NotFound desc = could not find container \"d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff\": container with ID starting with d187dede8fde06f774c57cfa7ece8654a00016ce4fd88570a0db4f5edab0f5ff not found: ID does not exist"
Sep 18 14:16:39.958117 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.958095    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"]
Sep 18 14:16:39.962670 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:39.962651    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6301-5c75f5976f-sfk5w"]
Sep 18 14:16:40.485996 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:40.485964    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ed52a62-3805-443f-bb1f-144c08b30c67" path="/var/lib/kubelet/pods/1ed52a62-3805-443f-bb1f-144c08b30c67/volumes"
Sep 18 14:16:40.486345 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:40.486331    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c2096e-f609-4ec1-9ae6-c4e607d35d67" path="/var/lib/kubelet/pods/43c2096e-f609-4ec1-9ae6-c4e607d35d67/volumes"
Sep 18 14:16:56.611913 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.611873    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"]
Sep 18 14:16:56.612324 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.612133    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6" podUID="359731fd-a9c3-4aa2-b652-1574aaabf48a" containerName="maas-api" containerID="cri-o://ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2" gracePeriod=30
Sep 18 14:16:56.718963 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.718915    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6" podUID="359731fd-a9c3-4aa2-b652-1574aaabf48a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.54:8443/health\": dial tcp 10.133.0.54:8443: connect: connection refused"
Sep 18 14:16:56.848804 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.848781    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:16:56.945656 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.945630    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359731fd-a9c3-4aa2-b652-1574aaabf48a-openshift-service-ca\") pod \"359731fd-a9c3-4aa2-b652-1574aaabf48a\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") "
Sep 18 14:16:56.945828 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.945738    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7psw\" (UniqueName: \"kubernetes.io/projected/359731fd-a9c3-4aa2-b652-1574aaabf48a-kube-api-access-t7psw\") pod \"359731fd-a9c3-4aa2-b652-1574aaabf48a\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") "
Sep 18 14:16:56.945828 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.945770    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-maas-api-tls\") pod \"359731fd-a9c3-4aa2-b652-1574aaabf48a\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") "
Sep 18 14:16:56.945955 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.945833    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-metrics-certs\") pod \"359731fd-a9c3-4aa2-b652-1574aaabf48a\" (UID: \"359731fd-a9c3-4aa2-b652-1574aaabf48a\") "
Sep 18 14:16:56.946091 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.946047    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359731fd-a9c3-4aa2-b652-1574aaabf48a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "359731fd-a9c3-4aa2-b652-1574aaabf48a" (UID: "359731fd-a9c3-4aa2-b652-1574aaabf48a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:16:56.948266 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.948242    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "359731fd-a9c3-4aa2-b652-1574aaabf48a" (UID: "359731fd-a9c3-4aa2-b652-1574aaabf48a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:16:56.948378 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.948317    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "359731fd-a9c3-4aa2-b652-1574aaabf48a" (UID: "359731fd-a9c3-4aa2-b652-1574aaabf48a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:16:56.948432 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:56.948410    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359731fd-a9c3-4aa2-b652-1574aaabf48a-kube-api-access-t7psw" (OuterVolumeSpecName: "kube-api-access-t7psw") pod "359731fd-a9c3-4aa2-b652-1574aaabf48a" (UID: "359731fd-a9c3-4aa2-b652-1574aaabf48a"). InnerVolumeSpecName "kube-api-access-t7psw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:57.013986 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.013950    2569 generic.go:356] "Generic (PLEG): container finished" podID="359731fd-a9c3-4aa2-b652-1574aaabf48a" containerID="ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2" exitCode=0
Sep 18 14:16:57.014172 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.014002    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"
Sep 18 14:16:57.014172 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.014039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6" event={"ID":"359731fd-a9c3-4aa2-b652-1574aaabf48a","Type":"ContainerDied","Data":"ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2"}
Sep 18 14:16:57.014172 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.014122    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6" event={"ID":"359731fd-a9c3-4aa2-b652-1574aaabf48a","Type":"ContainerDied","Data":"b350af84b405a7422ee333cb7ce491ecc122c49c23ce052827845cb9528266e3"}
Sep 18 14:16:57.014172 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.014147    2569 scope.go:117] "RemoveContainer" containerID="ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2"
Sep 18 14:16:57.029764 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.029741    2569 scope.go:117] "RemoveContainer" containerID="ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2"
Sep 18 14:16:57.030053 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:16:57.030033    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2\": container with ID starting with ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2 not found: ID does not exist" containerID="ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2"
Sep 18 14:16:57.030127 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.030081    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2"} err="failed to get container status \"ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2\": rpc error: code = NotFound desc = could not find container \"ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2\": container with ID starting with ebd9c46c5960e45d5143bdc1710a71515c2d52eda0e43b8f340fdc2664f6fab2 not found: ID does not exist"
Sep 18 14:16:57.042260 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.042232    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"]
Sep 18 14:16:57.046269 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.046238    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-783872-765fc8674c-q94d6"]
Sep 18 14:16:57.046561 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.046543    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7psw\" (UniqueName: \"kubernetes.io/projected/359731fd-a9c3-4aa2-b652-1574aaabf48a-kube-api-access-t7psw\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:57.046638 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.046563    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:57.046638 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.046573    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359731fd-a9c3-4aa2-b652-1574aaabf48a-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:57.046638 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:57.046591    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359731fd-a9c3-4aa2-b652-1574aaabf48a-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:16:58.486332 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:16:58.486295    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="359731fd-a9c3-4aa2-b652-1574aaabf48a" path="/var/lib/kubelet/pods/359731fd-a9c3-4aa2-b652-1574aaabf48a/volumes"
Sep 18 14:17:14.708340 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:14.708257    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"]
Sep 18 14:17:14.708779 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:14.708474    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg" podUID="f1dc29cc-a4f3-4646-9e96-28b245147efd" containerName="payload-pre-processing" containerID="cri-o://b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526" gracePeriod=30
Sep 18 14:17:14.960825 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:14.960765    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:17:15.100128 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.100103    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7p5x\" (UniqueName: \"kubernetes.io/projected/f1dc29cc-a4f3-4646-9e96-28b245147efd-kube-api-access-k7p5x\") pod \"f1dc29cc-a4f3-4646-9e96-28b245147efd\" (UID: \"f1dc29cc-a4f3-4646-9e96-28b245147efd\") "
Sep 18 14:17:15.100310 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.100169    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dc29cc-a4f3-4646-9e96-28b245147efd-plugins-config-volume\") pod \"f1dc29cc-a4f3-4646-9e96-28b245147efd\" (UID: \"f1dc29cc-a4f3-4646-9e96-28b245147efd\") "
Sep 18 14:17:15.100590 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.100566    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1dc29cc-a4f3-4646-9e96-28b245147efd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f1dc29cc-a4f3-4646-9e96-28b245147efd" (UID: "f1dc29cc-a4f3-4646-9e96-28b245147efd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:17:15.102443 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.102410    2569 generic.go:356] "Generic (PLEG): container finished" podID="f1dc29cc-a4f3-4646-9e96-28b245147efd" containerID="b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526" exitCode=0
Sep 18 14:17:15.102556 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.102454    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"
Sep 18 14:17:15.102556 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.102499    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg" event={"ID":"f1dc29cc-a4f3-4646-9e96-28b245147efd","Type":"ContainerDied","Data":"b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526"}
Sep 18 14:17:15.102653 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.102597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg" event={"ID":"f1dc29cc-a4f3-4646-9e96-28b245147efd","Type":"ContainerDied","Data":"02a543947a058b489bbd22280b7ed3b568189981dfba3d413ea3898445e24e67"}
Sep 18 14:17:15.102653 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.102629    2569 scope.go:117] "RemoveContainer" containerID="b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526"
Sep 18 14:17:15.102731 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.102682    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dc29cc-a4f3-4646-9e96-28b245147efd-kube-api-access-k7p5x" (OuterVolumeSpecName: "kube-api-access-k7p5x") pod "f1dc29cc-a4f3-4646-9e96-28b245147efd" (UID: "f1dc29cc-a4f3-4646-9e96-28b245147efd"). InnerVolumeSpecName "kube-api-access-k7p5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:15.113220 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.113198    2569 scope.go:117] "RemoveContainer" containerID="b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526"
Sep 18 14:17:15.113518 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:17:15.113491    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526\": container with ID starting with b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526 not found: ID does not exist" containerID="b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526"
Sep 18 14:17:15.113590 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.113528    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526"} err="failed to get container status \"b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526\": rpc error: code = NotFound desc = could not find container \"b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526\": container with ID starting with b518308d3d464eed443c73b332d8cb076ff8a21228a987020203f8863eb7f526 not found: ID does not exist"
Sep 18 14:17:15.201798 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.201769    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7p5x\" (UniqueName: \"kubernetes.io/projected/f1dc29cc-a4f3-4646-9e96-28b245147efd-kube-api-access-k7p5x\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:17:15.201798 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.201795    2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dc29cc-a4f3-4646-9e96-28b245147efd-plugins-config-volume\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:17:15.423727 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.423697    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"]
Sep 18 14:17:15.427636 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:15.427613    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1a4b6d-9d4466fbb-cqlvg"]
Sep 18 14:17:16.485636 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:16.485606    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1dc29cc-a4f3-4646-9e96-28b245147efd" path="/var/lib/kubelet/pods/f1dc29cc-a4f3-4646-9e96-28b245147efd/volumes"
Sep 18 14:17:26.210548 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.210516    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bd575b874-dd4ht"]
Sep 18 14:17:26.211345 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.210766    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bd575b874-dd4ht" podUID="9ea6c357-f700-4d2b-9cab-28fe91f89cfa" containerName="manager" containerID="cri-o://d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9" gracePeriod=30
Sep 18 14:17:26.455724 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.455700    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 14:17:26.605158 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.605050    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-metrics-certs\") pod \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") "
Sep 18 14:17:26.605158 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.605137    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhtc6\" (UniqueName: \"kubernetes.io/projected/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-kube-api-access-rhtc6\") pod \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") "
Sep 18 14:17:26.605353 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.605163    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-webhook-cert\") pod \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\" (UID: \"9ea6c357-f700-4d2b-9cab-28fe91f89cfa\") "
Sep 18 14:17:26.607293 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.607263    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9ea6c357-f700-4d2b-9cab-28fe91f89cfa" (UID: "9ea6c357-f700-4d2b-9cab-28fe91f89cfa"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:26.607420 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.607294    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ea6c357-f700-4d2b-9cab-28fe91f89cfa" (UID: "9ea6c357-f700-4d2b-9cab-28fe91f89cfa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:26.607420 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.607354    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-kube-api-access-rhtc6" (OuterVolumeSpecName: "kube-api-access-rhtc6") pod "9ea6c357-f700-4d2b-9cab-28fe91f89cfa" (UID: "9ea6c357-f700-4d2b-9cab-28fe91f89cfa"). InnerVolumeSpecName "kube-api-access-rhtc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:26.706087 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.706035    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhtc6\" (UniqueName: \"kubernetes.io/projected/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-kube-api-access-rhtc6\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:17:26.706259 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.706102    2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-webhook-cert\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:17:26.706259 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:26.706116    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea6c357-f700-4d2b-9cab-28fe91f89cfa-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:17:27.159154 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:27.159121    2569 generic.go:356] "Generic (PLEG): container finished" podID="9ea6c357-f700-4d2b-9cab-28fe91f89cfa" containerID="d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9" exitCode=0
Sep 18 14:17:27.159341 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:27.159177    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bd575b874-dd4ht" event={"ID":"9ea6c357-f700-4d2b-9cab-28fe91f89cfa","Type":"ContainerDied","Data":"d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9"}
Sep 18 14:17:27.159341 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:27.159180    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bd575b874-dd4ht"
Sep 18 14:17:27.159341 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:27.159220    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bd575b874-dd4ht" event={"ID":"9ea6c357-f700-4d2b-9cab-28fe91f89cfa","Type":"ContainerDied","Data":"e8134c029a7bea16c28a157f2b0110fa1fd636d2a21638598fd5b9ddd19e9a04"}
Sep 18 14:17:27.159341 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:27.159242    2569 scope.go:117] "RemoveContainer" containerID="d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9"
Sep 18 14:17:27.168890 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:27.168872    2569 scope.go:117] "RemoveContainer" containerID="d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9"
Sep 18 14:17:27.169188 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:17:27.169171    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9\": container with ID starting with d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9 not found: ID does not exist" containerID="d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9"
Sep 18 14:17:27.169245 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:27.169198    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9"} err="failed to get container status \"d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9\": rpc error: code = NotFound desc = could not find container \"d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9\": container with ID starting with d18a093fb499833d2b86de67cc757876161beb79717bf34daf8a13fb44503eb9 not found: ID does not exist"
Sep 18 14:17:27.181050 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:27.181026    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bd575b874-dd4ht"]
Sep 18 14:17:27.185286 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:27.185263    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bd575b874-dd4ht"]
Sep 18 14:17:28.490433 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:17:28.490401    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea6c357-f700-4d2b-9cab-28fe91f89cfa" path="/var/lib/kubelet/pods/9ea6c357-f700-4d2b-9cab-28fe91f89cfa/volumes"
Sep 18 14:18:27.992994 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.992956    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bd575b874-gmlws"]
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993290    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f1fcb61-15a1-493b-af21-6198786c17bd" containerName="maas-api"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993300    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f1fcb61-15a1-493b-af21-6198786c17bd" containerName="maas-api"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993312    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993317    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993323    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c2096e-f609-4ec1-9ae6-c4e607d35d67" containerName="payload-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993329    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c2096e-f609-4ec1-9ae6-c4e607d35d67" containerName="payload-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993337    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e729f446-d6f7-4846-a93f-7c5d58148515" containerName="payload-pre-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993342    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e729f446-d6f7-4846-a93f-7c5d58148515" containerName="payload-pre-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993350    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed52a62-3805-443f-bb1f-144c08b30c67" containerName="maas-api"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993355    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed52a62-3805-443f-bb1f-144c08b30c67" containerName="maas-api"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993362    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cee5a6e5-40d1-4d53-96a9-a190b3bed501" containerName="payload-pre-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993366    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cee5a6e5-40d1-4d53-96a9-a190b3bed501" containerName="payload-pre-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993382    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359731fd-a9c3-4aa2-b652-1574aaabf48a" containerName="maas-api"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993386    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="359731fd-a9c3-4aa2-b652-1574aaabf48a" containerName="maas-api"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993394    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea6c357-f700-4d2b-9cab-28fe91f89cfa" containerName="manager"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993399    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea6c357-f700-4d2b-9cab-28fe91f89cfa" containerName="manager"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993404    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1dc29cc-a4f3-4646-9e96-28b245147efd" containerName="payload-pre-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993409    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1dc29cc-a4f3-4646-9e96-28b245147efd" containerName="payload-pre-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993454    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f1fcb61-15a1-493b-af21-6198786c17bd" containerName="maas-api"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993461    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993467    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c2096e-f609-4ec1-9ae6-c4e607d35d67" containerName="payload-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993474    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e729f446-d6f7-4846-a93f-7c5d58148515" containerName="payload-pre-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993479    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cee5a6e5-40d1-4d53-96a9-a190b3bed501" containerName="payload-pre-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993484    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="359731fd-a9c3-4aa2-b652-1574aaabf48a" containerName="maas-api"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993490    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea6c357-f700-4d2b-9cab-28fe91f89cfa" containerName="manager"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993494    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1dc29cc-a4f3-4646-9e96-28b245147efd" containerName="payload-pre-processing"
Sep 18 14:18:27.993511 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.993499    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ed52a62-3805-443f-bb1f-144c08b30c67" containerName="maas-api"
Sep 18 14:18:27.996469 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.996453    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:27.999037 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.999018    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 14:18:27.999283 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:27.999265    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 14:18:28.000454 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.000428    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 14:18:28.000562 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.000442    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9kgv2\""
Sep 18 14:18:28.005795 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.005768    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bd575b874-gmlws"]
Sep 18 14:18:28.131033 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.131002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-metrics-certs\") pod \"maas-controller-5bd575b874-gmlws\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") " pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:28.131228 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.131061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvwtf\" (UniqueName: \"kubernetes.io/projected/fba3e205-713c-42ba-8762-fdf4618db081-kube-api-access-wvwtf\") pod \"maas-controller-5bd575b874-gmlws\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") " pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:28.131228 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.131161    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-webhook-cert\") pod \"maas-controller-5bd575b874-gmlws\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") " pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:28.232376 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.232345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvwtf\" (UniqueName: \"kubernetes.io/projected/fba3e205-713c-42ba-8762-fdf4618db081-kube-api-access-wvwtf\") pod \"maas-controller-5bd575b874-gmlws\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") " pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:28.232540 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.232408    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-webhook-cert\") pod \"maas-controller-5bd575b874-gmlws\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") " pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:28.232540 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.232438    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-metrics-certs\") pod \"maas-controller-5bd575b874-gmlws\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") " pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:28.235150 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.235119    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-webhook-cert\") pod \"maas-controller-5bd575b874-gmlws\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") " pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:28.235271 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.235119    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-metrics-certs\") pod \"maas-controller-5bd575b874-gmlws\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") " pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:28.242106 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.242064    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvwtf\" (UniqueName: \"kubernetes.io/projected/fba3e205-713c-42ba-8762-fdf4618db081-kube-api-access-wvwtf\") pod \"maas-controller-5bd575b874-gmlws\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") " pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:28.307696 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.307619    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:28.582064 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.581981    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bd575b874-gmlws"]
Sep 18 14:18:28.585041 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:18:28.585017    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba3e205_713c_42ba_8762_fdf4618db081.slice/crio-f24ad16753c356ff827891abd517f91da9965ea4cbd1f3a4f062acdd02444651 WatchSource:0}: Error finding container f24ad16753c356ff827891abd517f91da9965ea4cbd1f3a4f062acdd02444651: Status 404 returned error can't find the container with id f24ad16753c356ff827891abd517f91da9965ea4cbd1f3a4f062acdd02444651
Sep 18 14:18:28.586290 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:28.586270    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:18:29.444555 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:29.444512    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bd575b874-gmlws" event={"ID":"fba3e205-713c-42ba-8762-fdf4618db081","Type":"ContainerStarted","Data":"719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209"}
Sep 18 14:18:29.445054 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:29.444610    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bd575b874-gmlws" event={"ID":"fba3e205-713c-42ba-8762-fdf4618db081","Type":"ContainerStarted","Data":"f24ad16753c356ff827891abd517f91da9965ea4cbd1f3a4f062acdd02444651"}
Sep 18 14:18:29.445054 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:29.444660    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:29.462488 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:29.462442    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bd575b874-gmlws" podStartSLOduration=1.7616022340000002 podStartE2EDuration="2.462426967s" totalImagesPullingTime="700.824733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:29.46171755 +0000 UTC m=+2051.364173735" watchObservedRunningTime="2026-09-18 14:18:29.462426967 +0000 UTC m=+2051.364883153"
Sep 18 14:18:40.451128 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:40.451033    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:49.884434 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:49.884399    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bd575b874-gmlws"]
Sep 18 14:18:49.884875 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:49.884632    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bd575b874-gmlws" podUID="fba3e205-713c-42ba-8762-fdf4618db081" containerName="manager" containerID="cri-o://719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209" gracePeriod=30
Sep 18 14:18:50.132731 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.132708    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:50.234541 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.234503    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvwtf\" (UniqueName: \"kubernetes.io/projected/fba3e205-713c-42ba-8762-fdf4618db081-kube-api-access-wvwtf\") pod \"fba3e205-713c-42ba-8762-fdf4618db081\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") "
Sep 18 14:18:50.234700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.234603    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-metrics-certs\") pod \"fba3e205-713c-42ba-8762-fdf4618db081\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") "
Sep 18 14:18:50.234700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.234690    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-webhook-cert\") pod \"fba3e205-713c-42ba-8762-fdf4618db081\" (UID: \"fba3e205-713c-42ba-8762-fdf4618db081\") "
Sep 18 14:18:50.236733 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.236703    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fba3e205-713c-42ba-8762-fdf4618db081" (UID: "fba3e205-713c-42ba-8762-fdf4618db081"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:18:50.236851 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.236702    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba3e205-713c-42ba-8762-fdf4618db081-kube-api-access-wvwtf" (OuterVolumeSpecName: "kube-api-access-wvwtf") pod "fba3e205-713c-42ba-8762-fdf4618db081" (UID: "fba3e205-713c-42ba-8762-fdf4618db081"). InnerVolumeSpecName "kube-api-access-wvwtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:18:50.236900 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.236843    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fba3e205-713c-42ba-8762-fdf4618db081" (UID: "fba3e205-713c-42ba-8762-fdf4618db081"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:18:50.336215 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.336189    2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-webhook-cert\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:18:50.336215 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.336214    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvwtf\" (UniqueName: \"kubernetes.io/projected/fba3e205-713c-42ba-8762-fdf4618db081-kube-api-access-wvwtf\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:18:50.336393 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.336231    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fba3e205-713c-42ba-8762-fdf4618db081-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:18:50.540188 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.540156    2569 generic.go:356] "Generic (PLEG): container finished" podID="fba3e205-713c-42ba-8762-fdf4618db081" containerID="719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209" exitCode=0
Sep 18 14:18:50.540334 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.540201    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bd575b874-gmlws"
Sep 18 14:18:50.540334 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.540241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bd575b874-gmlws" event={"ID":"fba3e205-713c-42ba-8762-fdf4618db081","Type":"ContainerDied","Data":"719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209"}
Sep 18 14:18:50.540334 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.540290    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bd575b874-gmlws" event={"ID":"fba3e205-713c-42ba-8762-fdf4618db081","Type":"ContainerDied","Data":"f24ad16753c356ff827891abd517f91da9965ea4cbd1f3a4f062acdd02444651"}
Sep 18 14:18:50.540334 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.540313    2569 scope.go:117] "RemoveContainer" containerID="719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209"
Sep 18 14:18:50.548732 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.548712    2569 scope.go:117] "RemoveContainer" containerID="719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209"
Sep 18 14:18:50.549022 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:18:50.549004    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209\": container with ID starting with 719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209 not found: ID does not exist" containerID="719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209"
Sep 18 14:18:50.549121 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.549034    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209"} err="failed to get container status \"719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209\": rpc error: code = NotFound desc = could not find container \"719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209\": container with ID starting with 719f0c7c2c9ec984fc1f2422aedd332c2094560c4b26529bc2d83b17ff228209 not found: ID does not exist"
Sep 18 14:18:50.559350 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.559327    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bd575b874-gmlws"]
Sep 18 14:18:50.563415 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:50.563393    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bd575b874-gmlws"]
Sep 18 14:18:51.323692 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.323659    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bd575b874-dwqjm"]
Sep 18 14:18:51.324049 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.323961    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy"
Sep 18 14:18:51.324049 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.323972    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy"
Sep 18 14:18:51.324049 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.323992    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba3e205-713c-42ba-8762-fdf4618db081" containerName="manager"
Sep 18 14:18:51.324049 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.323998    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba3e205-713c-42ba-8762-fdf4618db081" containerName="manager"
Sep 18 14:18:51.324241 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.324053    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fba3e205-713c-42ba-8762-fdf4618db081" containerName="manager"
Sep 18 14:18:51.324241 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.324062    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="43998e37-a725-46d1-881c-721ac526aff2" containerName="istio-proxy"
Sep 18 14:18:51.328265 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.328249    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.332524 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.332501    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 14:18:51.332654 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.332554    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9kgv2\""
Sep 18 14:18:51.332654 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.332501    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 14:18:51.332654 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.332511    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 14:18:51.337463 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.337440    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bd575b874-dwqjm"]
Sep 18 14:18:51.444655 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.444623    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntcz8\" (UniqueName: \"kubernetes.io/projected/bbf316f6-d5ee-46ac-9e6c-990007b37aef-kube-api-access-ntcz8\") pod \"maas-controller-5bd575b874-dwqjm\" (UID: \"bbf316f6-d5ee-46ac-9e6c-990007b37aef\") " pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.444813 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.444665    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbf316f6-d5ee-46ac-9e6c-990007b37aef-webhook-cert\") pod \"maas-controller-5bd575b874-dwqjm\" (UID: \"bbf316f6-d5ee-46ac-9e6c-990007b37aef\") " pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.444813 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.444743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf316f6-d5ee-46ac-9e6c-990007b37aef-metrics-certs\") pod \"maas-controller-5bd575b874-dwqjm\" (UID: \"bbf316f6-d5ee-46ac-9e6c-990007b37aef\") " pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.545260 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.545227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntcz8\" (UniqueName: \"kubernetes.io/projected/bbf316f6-d5ee-46ac-9e6c-990007b37aef-kube-api-access-ntcz8\") pod \"maas-controller-5bd575b874-dwqjm\" (UID: \"bbf316f6-d5ee-46ac-9e6c-990007b37aef\") " pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.545438 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.545283    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbf316f6-d5ee-46ac-9e6c-990007b37aef-webhook-cert\") pod \"maas-controller-5bd575b874-dwqjm\" (UID: \"bbf316f6-d5ee-46ac-9e6c-990007b37aef\") " pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.545438 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.545330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf316f6-d5ee-46ac-9e6c-990007b37aef-metrics-certs\") pod \"maas-controller-5bd575b874-dwqjm\" (UID: \"bbf316f6-d5ee-46ac-9e6c-990007b37aef\") " pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.548091 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.548042    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbf316f6-d5ee-46ac-9e6c-990007b37aef-webhook-cert\") pod \"maas-controller-5bd575b874-dwqjm\" (UID: \"bbf316f6-d5ee-46ac-9e6c-990007b37aef\") " pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.548226 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.548102    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf316f6-d5ee-46ac-9e6c-990007b37aef-metrics-certs\") pod \"maas-controller-5bd575b874-dwqjm\" (UID: \"bbf316f6-d5ee-46ac-9e6c-990007b37aef\") " pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.554397 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.554378    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntcz8\" (UniqueName: \"kubernetes.io/projected/bbf316f6-d5ee-46ac-9e6c-990007b37aef-kube-api-access-ntcz8\") pod \"maas-controller-5bd575b874-dwqjm\" (UID: \"bbf316f6-d5ee-46ac-9e6c-990007b37aef\") " pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.638450 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.638370    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:51.909057 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:51.909032    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bd575b874-dwqjm"]
Sep 18 14:18:51.910445 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:18:51.910418    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf316f6_d5ee_46ac_9e6c_990007b37aef.slice/crio-9adb9274273aff00479917952977258f9dd4561f9dc9ce6b7810d1ac0d80381a WatchSource:0}: Error finding container 9adb9274273aff00479917952977258f9dd4561f9dc9ce6b7810d1ac0d80381a: Status 404 returned error can't find the container with id 9adb9274273aff00479917952977258f9dd4561f9dc9ce6b7810d1ac0d80381a
Sep 18 14:18:52.486268 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:52.486234    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fba3e205-713c-42ba-8762-fdf4618db081" path="/var/lib/kubelet/pods/fba3e205-713c-42ba-8762-fdf4618db081/volumes"
Sep 18 14:18:52.552711 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:52.552676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bd575b874-dwqjm" event={"ID":"bbf316f6-d5ee-46ac-9e6c-990007b37aef","Type":"ContainerStarted","Data":"0b43d37efd644611131127696c135e112746fd0846a549672de9dc4769d1f9b0"}
Sep 18 14:18:52.552884 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:52.552735    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bd575b874-dwqjm" event={"ID":"bbf316f6-d5ee-46ac-9e6c-990007b37aef","Type":"ContainerStarted","Data":"9adb9274273aff00479917952977258f9dd4561f9dc9ce6b7810d1ac0d80381a"}
Sep 18 14:18:52.552884 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:52.552762    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:18:52.571716 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:18:52.571625    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bd575b874-dwqjm" podStartSLOduration=1.173202979 podStartE2EDuration="1.571611574s" totalImagesPullingTime="398.408595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:52.569854336 +0000 UTC m=+2074.472310521" watchObservedRunningTime="2026-09-18 14:18:52.571611574 +0000 UTC m=+2074.474067761"
Sep 18 14:19:03.559761 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:19:03.559731    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bd575b874-dwqjm"
Sep 18 14:19:18.642011 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:19:18.641983    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 14:19:18.643003 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:19:18.642984    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 14:22:09.393725 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.393694    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"]
Sep 18 14:22:09.396893 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.396878    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.399751 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.399729    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-16aa6981-kserve-self-signed-certs\""
Sep 18 14:22:09.420101 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.414255    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"]
Sep 18 14:22:09.486821 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.486791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfd357eb-94be-44d8-9afe-852b2fab817e-tls-certs\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.487005 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.486842    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-home\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.487005 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.486913    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-dshm\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.487005 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.486965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-tmp-dir\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.487005 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.486991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2kv\" (UniqueName: \"kubernetes.io/projected/bfd357eb-94be-44d8-9afe-852b2fab817e-kube-api-access-tb2kv\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.487235 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.487022    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-model-cache\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.587943 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.587909    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-tmp-dir\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.587943 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.587947    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2kv\" (UniqueName: \"kubernetes.io/projected/bfd357eb-94be-44d8-9afe-852b2fab817e-kube-api-access-tb2kv\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.588181 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.587971    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-model-cache\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.588181 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.588029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfd357eb-94be-44d8-9afe-852b2fab817e-tls-certs\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.588181 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.588082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-home\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.588181 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.588134    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-dshm\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.588400 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.588381    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-tmp-dir\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.588460 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.588437    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-model-cache\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.588509 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.588443    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-home\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.590581 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.590564    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-dshm\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.590700 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.590683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfd357eb-94be-44d8-9afe-852b2fab817e-tls-certs\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.599174 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.599142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2kv\" (UniqueName: \"kubernetes.io/projected/bfd357eb-94be-44d8-9afe-852b2fab817e-kube-api-access-tb2kv\") pod \"e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") " pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.706346 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.706324    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:09.976767 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:09.976744    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"]
Sep 18 14:22:09.978681 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:22:09.978653    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfd357eb_94be_44d8_9afe_852b2fab817e.slice/crio-772689ffc9e1cb2471de1db31511bb606258b48a33e30153c8a2a0ba11efe4bd WatchSource:0}: Error finding container 772689ffc9e1cb2471de1db31511bb606258b48a33e30153c8a2a0ba11efe4bd: Status 404 returned error can't find the container with id 772689ffc9e1cb2471de1db31511bb606258b48a33e30153c8a2a0ba11efe4bd
Sep 18 14:22:10.418337 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:10.418251    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb" event={"ID":"bfd357eb-94be-44d8-9afe-852b2fab817e","Type":"ContainerStarted","Data":"f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14"}
Sep 18 14:22:10.418337 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:10.418289    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb" event={"ID":"bfd357eb-94be-44d8-9afe-852b2fab817e","Type":"ContainerStarted","Data":"772689ffc9e1cb2471de1db31511bb606258b48a33e30153c8a2a0ba11efe4bd"}
Sep 18 14:22:10.437336 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:10.437271    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb" podStartSLOduration=1.43725616 podStartE2EDuration="1.43725616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:22:10.435855665 +0000 UTC m=+2272.338311851" watchObservedRunningTime="2026-09-18 14:22:10.43725616 +0000 UTC m=+2272.339712345"
Sep 18 14:22:19.706678 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:19.706648    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:19.707092 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:19.706690    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:19.719320 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:19.719289    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:20.472418 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:20.472382    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:43.102366 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:43.102336    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"]
Sep 18 14:22:58.122088 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.122007    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb" podUID="bfd357eb-94be-44d8-9afe-852b2fab817e" containerName="main" containerID="cri-o://f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14" gracePeriod=45
Sep 18 14:22:58.367401 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.367377    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:58.488992 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.488966    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-home\") pod \"bfd357eb-94be-44d8-9afe-852b2fab817e\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") "
Sep 18 14:22:58.489176 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489012    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-tmp-dir\") pod \"bfd357eb-94be-44d8-9afe-852b2fab817e\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") "
Sep 18 14:22:58.489176 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489034    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-dshm\") pod \"bfd357eb-94be-44d8-9afe-852b2fab817e\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") "
Sep 18 14:22:58.489176 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489050    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-model-cache\") pod \"bfd357eb-94be-44d8-9afe-852b2fab817e\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") "
Sep 18 14:22:58.489176 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489130    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb2kv\" (UniqueName: \"kubernetes.io/projected/bfd357eb-94be-44d8-9afe-852b2fab817e-kube-api-access-tb2kv\") pod \"bfd357eb-94be-44d8-9afe-852b2fab817e\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") "
Sep 18 14:22:58.489176 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489165    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfd357eb-94be-44d8-9afe-852b2fab817e-tls-certs\") pod \"bfd357eb-94be-44d8-9afe-852b2fab817e\" (UID: \"bfd357eb-94be-44d8-9afe-852b2fab817e\") "
Sep 18 14:22:58.489448 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489271    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-home" (OuterVolumeSpecName: "home") pod "bfd357eb-94be-44d8-9afe-852b2fab817e" (UID: "bfd357eb-94be-44d8-9afe-852b2fab817e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:58.489448 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489290    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bfd357eb-94be-44d8-9afe-852b2fab817e" (UID: "bfd357eb-94be-44d8-9afe-852b2fab817e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:58.489448 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489391    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bfd357eb-94be-44d8-9afe-852b2fab817e" (UID: "bfd357eb-94be-44d8-9afe-852b2fab817e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:58.489553 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489523    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-home\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:22:58.489553 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489538    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-tmp-dir\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:22:58.489553 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.489547    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-model-cache\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:22:58.491510 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.491484    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-dshm" (OuterVolumeSpecName: "dshm") pod "bfd357eb-94be-44d8-9afe-852b2fab817e" (UID: "bfd357eb-94be-44d8-9afe-852b2fab817e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:58.491637 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.491520    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd357eb-94be-44d8-9afe-852b2fab817e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bfd357eb-94be-44d8-9afe-852b2fab817e" (UID: "bfd357eb-94be-44d8-9afe-852b2fab817e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:22:58.491637 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.491530    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd357eb-94be-44d8-9afe-852b2fab817e-kube-api-access-tb2kv" (OuterVolumeSpecName: "kube-api-access-tb2kv") pod "bfd357eb-94be-44d8-9afe-852b2fab817e" (UID: "bfd357eb-94be-44d8-9afe-852b2fab817e"). InnerVolumeSpecName "kube-api-access-tb2kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:22:58.590579 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.590544    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb2kv\" (UniqueName: \"kubernetes.io/projected/bfd357eb-94be-44d8-9afe-852b2fab817e-kube-api-access-tb2kv\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:22:58.590579 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.590578    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfd357eb-94be-44d8-9afe-852b2fab817e-tls-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:22:58.590579 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.590587    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfd357eb-94be-44d8-9afe-852b2fab817e-dshm\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:22:58.633166 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.633142    2569 generic.go:356] "Generic (PLEG): container finished" podID="bfd357eb-94be-44d8-9afe-852b2fab817e" containerID="f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14" exitCode=0
Sep 18 14:22:58.633283 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.633182    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"
Sep 18 14:22:58.633283 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.633218    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb" event={"ID":"bfd357eb-94be-44d8-9afe-852b2fab817e","Type":"ContainerDied","Data":"f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14"}
Sep 18 14:22:58.633283 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.633248    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb" event={"ID":"bfd357eb-94be-44d8-9afe-852b2fab817e","Type":"ContainerDied","Data":"772689ffc9e1cb2471de1db31511bb606258b48a33e30153c8a2a0ba11efe4bd"}
Sep 18 14:22:58.633283 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.633263    2569 scope.go:117] "RemoveContainer" containerID="f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14"
Sep 18 14:22:58.647848 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.647828    2569 scope.go:117] "RemoveContainer" containerID="f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14"
Sep 18 14:22:58.648174 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:22:58.648152    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14\": container with ID starting with f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14 not found: ID does not exist" containerID="f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14"
Sep 18 14:22:58.648243 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.648179    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14"} err="failed to get container status \"f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14\": rpc error: code = NotFound desc = could not find container \"f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14\": container with ID starting with f5dd43e11b14844162164c121ba8374d9055f7a781d51e765bc8abab8349de14 not found: ID does not exist"
Sep 18 14:22:58.658128 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.658108    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"]
Sep 18 14:22:58.665324 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:22:58.665304    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-16aa6981-kserve-ff6647ff5-k54mb"]
Sep 18 14:23:00.485298 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:00.485265    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfd357eb-94be-44d8-9afe-852b2fab817e" path="/var/lib/kubelet/pods/bfd357eb-94be-44d8-9afe-852b2fab817e/volumes"
Sep 18 14:23:01.233301 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.233268    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"]
Sep 18 14:23:01.233596 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.233584    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd357eb-94be-44d8-9afe-852b2fab817e" containerName="main"
Sep 18 14:23:01.233641 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.233598    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd357eb-94be-44d8-9afe-852b2fab817e" containerName="main"
Sep 18 14:23:01.233676 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.233656    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd357eb-94be-44d8-9afe-852b2fab817e" containerName="main"
Sep 18 14:23:01.238167 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.238134    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.251770 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.251748    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"]
Sep 18 14:23:01.312531 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.312500    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92xz\" (UniqueName: \"kubernetes.io/projected/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-kube-api-access-t92xz\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.312672 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.312561    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-metrics-certs\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.312672 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.312584    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-openshift-service-ca\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.312784 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.312674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-maas-api-tls\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.414195 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.414160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-metrics-certs\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.414379 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.414201    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-openshift-service-ca\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.414379 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.414345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-maas-api-tls\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.414506 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.414433    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92xz\" (UniqueName: \"kubernetes.io/projected/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-kube-api-access-t92xz\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.414826 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.414808    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-openshift-service-ca\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.416992 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.416971    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-metrics-certs\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.417198 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.417179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-maas-api-tls\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.422463 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.422437    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92xz\" (UniqueName: \"kubernetes.io/projected/e87e4e07-5aff-4f3b-8ce3-bddda87c87c7-kube-api-access-t92xz\") pod \"maas-api-78fd7758f9-wm6fj\" (UID: \"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7\") " pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.549330 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.549245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:01.825805 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:01.825779    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"]
Sep 18 14:23:01.827751 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:23:01.827722    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87e4e07_5aff_4f3b_8ce3_bddda87c87c7.slice/crio-a33af0bf564486c782fe1a9b97bad4d448483b374519e878b4e9f6cc4ee52846 WatchSource:0}: Error finding container a33af0bf564486c782fe1a9b97bad4d448483b374519e878b4e9f6cc4ee52846: Status 404 returned error can't find the container with id a33af0bf564486c782fe1a9b97bad4d448483b374519e878b4e9f6cc4ee52846
Sep 18 14:23:02.655693 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:02.655611    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj" event={"ID":"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7","Type":"ContainerStarted","Data":"56a7aacc2e7f59415eba6cd101dcc18e2de366f73796aa1acd677880ec821129"}
Sep 18 14:23:02.655693 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:02.655670    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj" event={"ID":"e87e4e07-5aff-4f3b-8ce3-bddda87c87c7","Type":"ContainerStarted","Data":"a33af0bf564486c782fe1a9b97bad4d448483b374519e878b4e9f6cc4ee52846"}
Sep 18 14:23:02.656128 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:02.655733    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:02.675419 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:02.675367    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj" podStartSLOduration=1.205207779 podStartE2EDuration="1.675355224s" totalImagesPullingTime="470.147445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:23:02.673102277 +0000 UTC m=+2324.575558461" watchObservedRunningTime="2026-09-18 14:23:02.675355224 +0000 UTC m=+2324.577811409"
Sep 18 14:23:08.664709 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:08.664683    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78fd7758f9-wm6fj"
Sep 18 14:23:16.011401 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:16.011368    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"]
Sep 18 14:23:16.011871 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:16.011624    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn" podUID="5b11d110-87d2-4e58-bf10-1f27550f0238" containerName="maas-api" containerID="cri-o://41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c" gracePeriod=30
Sep 18 14:23:17.251242 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.251215    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 14:23:17.348628 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.348565    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-metrics-certs\") pod \"5b11d110-87d2-4e58-bf10-1f27550f0238\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") "
Sep 18 14:23:17.348628 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.348611    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-maas-api-tls\") pod \"5b11d110-87d2-4e58-bf10-1f27550f0238\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") "
Sep 18 14:23:17.348822 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.348646    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9wgm\" (UniqueName: \"kubernetes.io/projected/5b11d110-87d2-4e58-bf10-1f27550f0238-kube-api-access-s9wgm\") pod \"5b11d110-87d2-4e58-bf10-1f27550f0238\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") "
Sep 18 14:23:17.348822 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.348668    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b11d110-87d2-4e58-bf10-1f27550f0238-openshift-service-ca\") pod \"5b11d110-87d2-4e58-bf10-1f27550f0238\" (UID: \"5b11d110-87d2-4e58-bf10-1f27550f0238\") "
Sep 18 14:23:17.349129 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.349099    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b11d110-87d2-4e58-bf10-1f27550f0238-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b11d110-87d2-4e58-bf10-1f27550f0238" (UID: "5b11d110-87d2-4e58-bf10-1f27550f0238"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:23:17.350846 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.350817    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b11d110-87d2-4e58-bf10-1f27550f0238" (UID: "5b11d110-87d2-4e58-bf10-1f27550f0238"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:17.350943 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.350856    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b11d110-87d2-4e58-bf10-1f27550f0238" (UID: "5b11d110-87d2-4e58-bf10-1f27550f0238"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:17.350943 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.350873    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b11d110-87d2-4e58-bf10-1f27550f0238-kube-api-access-s9wgm" (OuterVolumeSpecName: "kube-api-access-s9wgm") pod "5b11d110-87d2-4e58-bf10-1f27550f0238" (UID: "5b11d110-87d2-4e58-bf10-1f27550f0238"). InnerVolumeSpecName "kube-api-access-s9wgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:17.449489 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.449465    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-metrics-certs\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:23:17.449489 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.449486    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b11d110-87d2-4e58-bf10-1f27550f0238-maas-api-tls\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:23:17.449702 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.449497    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9wgm\" (UniqueName: \"kubernetes.io/projected/5b11d110-87d2-4e58-bf10-1f27550f0238-kube-api-access-s9wgm\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:23:17.449702 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.449507    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b11d110-87d2-4e58-bf10-1f27550f0238-openshift-service-ca\") on node \"ip-10-0-143-135.ec2.internal\" DevicePath \"\""
Sep 18 14:23:17.734679 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.734648    2569 generic.go:356] "Generic (PLEG): container finished" podID="5b11d110-87d2-4e58-bf10-1f27550f0238" containerID="41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c" exitCode=0
Sep 18 14:23:17.734809 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.734696    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"
Sep 18 14:23:17.734809 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.734736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn" event={"ID":"5b11d110-87d2-4e58-bf10-1f27550f0238","Type":"ContainerDied","Data":"41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c"}
Sep 18 14:23:17.734809 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.734791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn" event={"ID":"5b11d110-87d2-4e58-bf10-1f27550f0238","Type":"ContainerDied","Data":"6a1e427d0925181499f8318b45176521713a1ea3529e28c2ce020d6895835357"}
Sep 18 14:23:17.734933 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.734813    2569 scope.go:117] "RemoveContainer" containerID="41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c"
Sep 18 14:23:17.744397 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.744379    2569 scope.go:117] "RemoveContainer" containerID="41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c"
Sep 18 14:23:17.744692 ip-10-0-143-135 kubenswrapper[2569]: E0918 14:23:17.744658    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c\": container with ID starting with 41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c not found: ID does not exist" containerID="41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c"
Sep 18 14:23:17.744757 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.744705    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c"} err="failed to get container status \"41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c\": rpc error: code = NotFound desc = could not find container \"41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c\": container with ID starting with 41b3c5cc9208db2904c62b42b1a7a925ad0a9de9948ca7b5c3cdb4620459121c not found: ID does not exist"
Sep 18 14:23:17.757089 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.757055    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"]
Sep 18 14:23:17.763119 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:17.763099    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-859bfcbfc5-2nxnn"]
Sep 18 14:23:18.487270 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:18.487241    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b11d110-87d2-4e58-bf10-1f27550f0238" path="/var/lib/kubelet/pods/5b11d110-87d2-4e58-bf10-1f27550f0238/volumes"
Sep 18 14:23:37.718405 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:37.718370    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bd575b874-dwqjm_bbf316f6-d5ee-46ac-9e6c-990007b37aef/manager/0.log"
Sep 18 14:23:37.828664 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:37.828637    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xttwp_eb9f7b40-29d0-4d27-8b52-b1e14cd41efd/server/0.log"
Sep 18 14:23:38.328747 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:38.328721    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-978ccd68d-j6jwg_d848ed91-7685-4d82-a7cc-2717d05a00d1/copy-manifests/0.log"
Sep 18 14:23:38.340871 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:38.340851    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-978ccd68d-j6jwg_d848ed91-7685-4d82-a7cc-2717d05a00d1/manager/0.log"
Sep 18 14:23:38.807121 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:38.807094    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78fd7758f9-wm6fj_e87e4e07-5aff-4f3b-8ce3-bddda87c87c7/maas-api/0.log"
Sep 18 14:23:40.054188 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:40.054156    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-rtn8p_0390bcd3-c90d-40c7-951f-f7fd43e2dda4/manager/0.log"
Sep 18 14:23:40.173598 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:40.173573    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-dhgtv_db99daa1-02fc-4b79-be6a-132f09da2ed1/manager/0.log"
Sep 18 14:23:40.673786 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:40.673758    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-6smsk_24846b11-1e3d-44c2-aac6-81279410b544/manager/0.log"
Sep 18 14:23:41.155659 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:41.155631    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9lvdf_3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7/discovery/0.log"
Sep 18 14:23:41.382754 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:41.382724    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5c57f774-9wsc8_4e71f7b4-32a8-4775-894c-a9853de09d38/kube-auth-proxy/0.log"
Sep 18 14:23:41.841240 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:41.841210    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86c7b86bcb-4jjm6_9ddf2987-06c4-4c7e-a2a8-f89293cc0497/router/0.log"
Sep 18 14:23:42.177657 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:42.177568    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-tppvs_35583c22-c074-4bf7-bdeb-6af8abb2f354/main/0.log"
Sep 18 14:23:42.291345 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:42.291323    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-g2bjk_bfaf8547-6873-4ae9-b9a7-cbfded37c070/main/0.log"
Sep 18 14:23:51.929533 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:51.929504    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fkhh5_4b31645a-f2d0-44ba-aa15-0bc175cb4319/global-pull-secret-syncer/0.log"
Sep 18 14:23:52.054411 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:52.054385    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nvrgd_b6ddd8f8-7119-4e4a-aaf2-3247b824a726/konnectivity-agent/0.log"
Sep 18 14:23:52.129313 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:52.129289    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-135.ec2.internal_52751f6427256e897ca3f2a8285d32e0/haproxy/0.log"
Sep 18 14:23:56.995703 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:56.995673    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-rtn8p_0390bcd3-c90d-40c7-951f-f7fd43e2dda4/manager/0.log"
Sep 18 14:23:57.039803 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:57.039777    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-dhgtv_db99daa1-02fc-4b79-be6a-132f09da2ed1/manager/0.log"
Sep 18 14:23:57.199088 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:57.199047    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-6smsk_24846b11-1e3d-44c2-aac6-81279410b544/manager/0.log"
Sep 18 14:23:59.066465 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:59.066435    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w6fd7_dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6/node-exporter/0.log"
Sep 18 14:23:59.095664 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:59.095642    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w6fd7_dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6/kube-rbac-proxy/0.log"
Sep 18 14:23:59.118802 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:23:59.118784    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w6fd7_dfbcf0f6-df17-4423-9018-d5c8dcd7e3e6/init-textfile/0.log"
Sep 18 14:24:01.171324 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.171297    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-7tmp9_33518b78-9e45-487b-84e7-1a053f46930b/networking-console-plugin/0.log"
Sep 18 14:24:01.824545 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.824509    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"]
Sep 18 14:24:01.824858 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.824844    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b11d110-87d2-4e58-bf10-1f27550f0238" containerName="maas-api"
Sep 18 14:24:01.824858 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.824859    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b11d110-87d2-4e58-bf10-1f27550f0238" containerName="maas-api"
Sep 18 14:24:01.824959 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.824919    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b11d110-87d2-4e58-bf10-1f27550f0238" containerName="maas-api"
Sep 18 14:24:01.827964 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.827945    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.831923 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.831900    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8mptg\"/\"default-dockercfg-9mm44\""
Sep 18 14:24:01.832080 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.831900    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mptg\"/\"openshift-service-ca.crt\""
Sep 18 14:24:01.832080 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.831965    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mptg\"/\"kube-root-ca.crt\""
Sep 18 14:24:01.838254 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.838231    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"]
Sep 18 14:24:01.886019 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.885995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-proc\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.886173 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.886028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-lib-modules\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.886173 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.886150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsq8\" (UniqueName: \"kubernetes.io/projected/1f3be43a-aa8e-424e-91ea-c7722c0ae854-kube-api-access-wjsq8\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.886300 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.886185    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-podres\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.886300 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.886232    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-sys\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.987644 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.987625    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-sys\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.987775 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.987656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-proc\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.987775 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.987674    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-lib-modules\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.987775 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.987725    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsq8\" (UniqueName: \"kubernetes.io/projected/1f3be43a-aa8e-424e-91ea-c7722c0ae854-kube-api-access-wjsq8\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.987775 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.987744    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-sys\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.987775 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.987758    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-proc\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.987982 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.987745    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-podres\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.987982 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.987845    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-lib-modules\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.987982 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.987850    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f3be43a-aa8e-424e-91ea-c7722c0ae854-podres\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:01.995899 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:01.995871    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsq8\" (UniqueName: \"kubernetes.io/projected/1f3be43a-aa8e-424e-91ea-c7722c0ae854-kube-api-access-wjsq8\") pod \"perf-node-gather-daemonset-pp7bn\" (UID: \"1f3be43a-aa8e-424e-91ea-c7722c0ae854\") " pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:02.138714 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:02.138663    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:02.408331 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:02.408307    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"]
Sep 18 14:24:02.410581 ip-10-0-143-135 kubenswrapper[2569]: W0918 14:24:02.410555    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1f3be43a_aa8e_424e_91ea_c7722c0ae854.slice/crio-43a7474c8b7a5f8b478163ab93aa4f3eafaa4cae09dbd252e17cc42c96877684 WatchSource:0}: Error finding container 43a7474c8b7a5f8b478163ab93aa4f3eafaa4cae09dbd252e17cc42c96877684: Status 404 returned error can't find the container with id 43a7474c8b7a5f8b478163ab93aa4f3eafaa4cae09dbd252e17cc42c96877684
Sep 18 14:24:02.952086 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:02.952032    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn" event={"ID":"1f3be43a-aa8e-424e-91ea-c7722c0ae854","Type":"ContainerStarted","Data":"b3a90e673dfd2a36e422f31da49619b21fc34c4a5e8630f06f36a305c9266585"}
Sep 18 14:24:02.952253 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:02.952103    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn" event={"ID":"1f3be43a-aa8e-424e-91ea-c7722c0ae854","Type":"ContainerStarted","Data":"43a7474c8b7a5f8b478163ab93aa4f3eafaa4cae09dbd252e17cc42c96877684"}
Sep 18 14:24:02.952253 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:02.952125    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:02.968020 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:02.967976    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn" podStartSLOduration=1.967961326 podStartE2EDuration="1.967961326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 14:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:24:02.966848204 +0000 UTC m=+2384.869304392" watchObservedRunningTime="2026-09-18 14:24:02.967961326 +0000 UTC m=+2384.870417550"
Sep 18 14:24:03.075340 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:03.075321    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4c5m8_e582da33-0688-485d-90de-91ef855ccc3a/volume-data-source-validator/0.log"
Sep 18 14:24:04.204109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:04.204063    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8ksvp_07d38f5e-d217-4846-a81f-640571391363/dns/0.log"
Sep 18 14:24:04.225274 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:04.225249    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8ksvp_07d38f5e-d217-4846-a81f-640571391363/kube-rbac-proxy/0.log"
Sep 18 14:24:04.398930 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:04.398909    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ww9c2_ded613ac-9a58-46d0-9ca6-b66852d872cb/dns-node-resolver/0.log"
Sep 18 14:24:04.965688 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:04.965661    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7596bf68dd-lr7t4_48c4cebb-4fe4-42c8-9bb0-0336a139bfdb/registry/0.log"
Sep 18 14:24:04.985487 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:04.985461    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2lqjs_742dfd2a-3975-49f5-84f5-3bbf7298b399/node-ca/0.log"
Sep 18 14:24:06.399948 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:06.399916    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9lvdf_3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7/discovery/0.log"
Sep 18 14:24:06.447972 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:06.447947    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5c57f774-9wsc8_4e71f7b4-32a8-4775-894c-a9853de09d38/kube-auth-proxy/0.log"
Sep 18 14:24:06.581109 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:06.581064    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86c7b86bcb-4jjm6_9ddf2987-06c4-4c7e-a2a8-f89293cc0497/router/0.log"
Sep 18 14:24:07.480988 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:07.480957    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8n4vp_64efa29e-40f6-4f0f-99c5-8fcbb48b687a/serve-healthcheck-canary/0.log"
Sep 18 14:24:08.020767 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:08.020735    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-sfzlp_cc6c4e02-80c7-463b-8cca-be8f1588caee/insights-operator/0.log"
Sep 18 14:24:08.020767 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:08.020767    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-sfzlp_cc6c4e02-80c7-463b-8cca-be8f1588caee/insights-operator/1.log"
Sep 18 14:24:08.114481 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:08.114464    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5mkl4_d4d7f5a1-3d5c-446d-b9df-8cf78eed185a/kube-rbac-proxy/0.log"
Sep 18 14:24:08.138739 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:08.138714    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5mkl4_d4d7f5a1-3d5c-446d-b9df-8cf78eed185a/exporter/0.log"
Sep 18 14:24:08.161778 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:08.161754    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5mkl4_d4d7f5a1-3d5c-446d-b9df-8cf78eed185a/extractor/0.log"
Sep 18 14:24:08.966764 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:08.966736    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8mptg/perf-node-gather-daemonset-pp7bn"
Sep 18 14:24:10.726443 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:10.726376    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bd575b874-dwqjm_bbf316f6-d5ee-46ac-9e6c-990007b37aef/manager/0.log"
Sep 18 14:24:10.749335 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:10.749317    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xttwp_eb9f7b40-29d0-4d27-8b52-b1e14cd41efd/server/0.log"
Sep 18 14:24:11.034724 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:11.034648    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-978ccd68d-j6jwg_d848ed91-7685-4d82-a7cc-2717d05a00d1/manager/0.log"
Sep 18 14:24:11.052823 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:11.052803    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-978ccd68d-j6jwg_d848ed91-7685-4d82-a7cc-2717d05a00d1/copy-manifests/0.log"
Sep 18 14:24:13.188141 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:13.188111    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9lvdf_3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7/discovery/0.log"
Sep 18 14:24:13.216403 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:13.216376    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5c57f774-9wsc8_4e71f7b4-32a8-4775-894c-a9853de09d38/kube-auth-proxy/0.log"
Sep 18 14:24:13.317897 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:13.317881    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86c7b86bcb-4jjm6_9ddf2987-06c4-4c7e-a2a8-f89293cc0497/router/0.log"
Sep 18 14:24:15.374000 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:15.373972    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9lvdf_3f6a0d3c-03cc-4ee1-893e-4d2ce90d5ab7/discovery/0.log"
Sep 18 14:24:15.416144 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:15.416123    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5c57f774-9wsc8_4e71f7b4-32a8-4775-894c-a9853de09d38/kube-auth-proxy/0.log"
Sep 18 14:24:15.512791 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:15.512769    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86c7b86bcb-4jjm6_9ddf2987-06c4-4c7e-a2a8-f89293cc0497/router/0.log"
Sep 18 14:24:18.239678 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:18.239645    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-9rn5l_f031d87d-90bc-46f0-b1e1-9cc72ef38e3b/kube-storage-version-migrator-operator/1.log"
Sep 18 14:24:18.240537 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:18.240500    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-9rn5l_f031d87d-90bc-46f0-b1e1-9cc72ef38e3b/kube-storage-version-migrator-operator/0.log"
Sep 18 14:24:18.669485 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:18.669412    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"
Sep 18 14:24:18.671856 ip-10-0-143-135 kubenswrapper[2569]: I0918 14:24:18.671835    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wcb2h_2fbbca64-d3f9-42c6-8175-94b778fbdcb7/ovn-acl-logging/0.log"