Sep 15 18:48:49.489717 ip-10-0-135-221 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 18:48:49.489733 ip-10-0-135-221 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 18:48:49.489743 ip-10-0-135-221 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 18:48:49.490122 ip-10-0-135-221 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 18:48:59.528388 ip-10-0-135-221 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 18:48:59.528409 ip-10-0-135-221 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bfdff11ef1b2492bbe7b7229670cf019 --
Sep 15 18:50:49.350113 ip-10-0-135-221 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 18:50:49.778217 ip-10-0-135-221 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:49.778217 ip-10-0-135-221 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 18:50:49.778217 ip-10-0-135-221 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:49.778217 ip-10-0-135-221 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 18:50:49.778217 ip-10-0-135-221 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:49.781415 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.781332    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 18:50:49.786186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786171    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:49.786186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786186    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786190    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786195    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786199    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786202    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786205    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786209    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786211    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786215    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786217    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786220    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786225    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786229    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786232    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786234    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786237    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786240    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786242    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786245    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:49.786256 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786247    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786250    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786254    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786256    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786259    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786262    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786265    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786268    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786270    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786273    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786276    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786278    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786281    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786284    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786286    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786289    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786291    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786294    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786297    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:49.786811 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786299    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786302    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786304    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786307    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786309    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786312    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786314    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786317    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786319    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786322    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786324    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786327    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786329    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786333    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786335    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786339    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786342    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786345    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786347    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786350    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:49.787290 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786352    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786355    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786358    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786360    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786363    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786365    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786368    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786370    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786373    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786375    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786378    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786381    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786384    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786386    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786389    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786392    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786394    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786397    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786400    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786402    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:49.787783 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786404    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786407    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786409    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786412    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786414    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786417    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786419    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786422    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786425    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786429    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786433    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786437    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786440    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786443    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786445    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786448    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786451    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786454    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786457    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:49.788286 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.786459    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787215    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787222    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787225    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787228    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787231    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787233    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787237    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787240    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787243    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787245    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787248    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787250    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787253    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787256    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787260    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787263    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787266    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787269    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:49.788793 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787271    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787276    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787279    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787281    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787284    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787287    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787290    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787293    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787296    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787298    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787301    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787304    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787306    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787309    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787312    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787314    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787317    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787319    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787322    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787325    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:49.789285 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787328    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787330    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787333    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787335    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787338    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787341    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787343    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787346    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787348    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787351    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787354    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787358    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787360    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787364    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787367    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787370    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787373    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787376    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787378    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787381    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:49.789778 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787383    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787386    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787388    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787391    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787393    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787395    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787398    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787401    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787403    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787405    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787408    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787410    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787413    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787415    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787418    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787421    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787423    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787426    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787428    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787431    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:49.790345 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787433    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787436    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787438    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787441    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787443    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787446    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787449    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787452    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787454    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787457    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787460    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787463    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787466    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787470    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787472    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787475    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787478    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787481    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787483    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787486    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:49.791077 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787488    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787854    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787859    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787862    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787865    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787868    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787871    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787874    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787876    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787879    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787882    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787884    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787887    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787890    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787892    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787895    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787897    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787900    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787903    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787906    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:49.791610 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787910    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787914    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787918    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787922    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787924    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787928    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787932    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787935    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787938    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787940    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787943    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787945    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787951    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787954    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787956    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787959    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787962    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787965    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787967    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:49.792148 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787970    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787973    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787975    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787978    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787980    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787983    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787985    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787988    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787990    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787993    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787996    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.787999    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788001    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788005    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788008    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788010    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788014    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788016    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788019    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:49.792633 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788021    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788024    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788026    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788029    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788031    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788034    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788038    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788040    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788043    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788045    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788047    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788050    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788052    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788055    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788057    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788060    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788062    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788065    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788068    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788070    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:49.793143 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788073    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788075    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788078    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788080    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788083    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788086    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788089    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788091    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788094    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788097    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788100    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788102    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788105    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788107    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788110    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788112    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788115    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788117    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788122    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:49.793654 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788124    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788127    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788129    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788199    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788235    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788241    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788245    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788249    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788252    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788256    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788261    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788264    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788267    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788270    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788274    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788277    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788280    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788283    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788286    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788290    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788293    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788298    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788301    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788304    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 18:50:49.794199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788308    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788311    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788315    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788318    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788321    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788324    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788327    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788331    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788334    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788337    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788340    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788344    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788347    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788350    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788353    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788356    2578 flags.go:64] FLAG: --enable-server="true"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788359    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788364    2578 flags.go:64] FLAG: --event-burst="100"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788367    2578 flags.go:64] FLAG: --event-qps="50"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788370    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788372    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788375    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788379    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788382    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788385    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 18:50:49.794969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788389    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788392    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788395    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788398    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788401    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788404    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788407    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788410    2578 flags.go:64] FLAG: --feature-gates=""
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788414    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788417    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788420    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788423    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788426    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788430    2578 flags.go:64] FLAG: --help="false"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788433    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-221.ec2.internal"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788436    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788440    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788443    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788446    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788449    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788452    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788455    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788458    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 18:50:49.795582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788461    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788464    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788467    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788470    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788473    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788475    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788478    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788481    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788486    2578 flags.go:64] FLAG: --lock-file=""
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788489    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788492    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788495    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788500    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788503    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788505    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788508    2578 flags.go:64] FLAG: --logging-format="text"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788511    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788515    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788518    2578 flags.go:64] FLAG: --manifest-url=""
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788521    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788525    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788528    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788532    2578 flags.go:64] FLAG: --max-pods="110"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788536    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788539    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 18:50:49.796216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788542    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788545    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788548    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788551    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788554    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788561    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788564    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788567    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788570    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788573    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788578    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788581    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788584    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788587    2578 flags.go:64] FLAG: --port="10250"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788590    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788593    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0407103e51e9b9119"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788598    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788601    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788604    2578 flags.go:64] FLAG: --register-node="true"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788606    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788611    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788613    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788616    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788619    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788623    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 18:50:49.796855 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788626    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788630    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788634    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788637    2578 flags.go:64] FLAG: --runonce="false"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788639    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788643    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788646    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788649    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788651    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788655    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788658    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788661    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788663    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788666    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788669    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788672    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788675    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788678    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788681    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788686    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788688    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788691    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788695    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788700    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788702    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 18:50:49.797515 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788705    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788708    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788711    2578 flags.go:64] FLAG: --v="2"
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788715    2578 flags.go:64] FLAG: --version="false"
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788719    2578 flags.go:64] FLAG: --vmodule=""
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788722    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.788725    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788850    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788854    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788858    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788861    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788863    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788868    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788870    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788873    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788876    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788879    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788881    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788884    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788887    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788890    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:49.798186 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788892    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788895    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788897    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788900    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788903    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788905    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788908    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788910    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788913    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788916    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788919    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788922    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788924    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788929    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788948    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788952    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788956    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788959    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788963    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:49.798703 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788967    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788970    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788973    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788977    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788982    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788986    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788989    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788992    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788994    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.788997    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789000    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789002    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789005    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789007    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789010    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789013    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789015    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789018    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789030    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789033    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:49.799268 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789035    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789038    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789042    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789045    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789047    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789050    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789052    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789055    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789057    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789060    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789063    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789065    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789068    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789070    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789073    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789076    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789080    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789083    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789086    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789089    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:49.799789 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789092    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789094    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789097    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789100    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789102    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789105    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789107    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789110    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789113    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789115    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789118    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789120    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789123    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789126    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789131    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789134    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789136    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789139    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789142    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789144    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:49.800307 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789147    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789149    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789152    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789154    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789157    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.789159    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.789872    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.800561    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.800578    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.800598    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.800606    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800658    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800663    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800666    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800670    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:49.800820 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800673    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800677    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800680    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800683    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800686    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800689    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800692    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800695    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800698    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800700    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800703    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800706    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800709    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800711    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800714    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800716    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800725    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800728    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800730    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:49.801208 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800733    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800736    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800738    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800741    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800744    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800747    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800750    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800753    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800755    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800758    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800761    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800763    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800766    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800769    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800771    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800774    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800776    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800779    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800782    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800784    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:49.801670 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800787    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800789    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800792    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800810    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800814    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800818    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800822    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800826    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800829    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800831    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800834    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800837    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800839    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800843    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800845    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800848    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800851    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800854    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800857    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800860    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:49.802200 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800863    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800866    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800868    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800871    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800874    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800877    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800879    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800882    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800885    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800887    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800890    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800893    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800897    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800901    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800904    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800908    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800912    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800915    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800918    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:49.802690 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800920    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800923    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800927    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800930    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800933    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800936    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800940    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800943    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800946    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800950    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800952    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800955    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800958    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800960    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800963    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800966    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:49.803180 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.800968    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.800973    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801079    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801084    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801087    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801090    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801093    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801096    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801099    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801102    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801104    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801107    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801110    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801113    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:49.803574 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801116    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801119    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801122    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801124    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801127    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801129    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801132    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801135    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801137    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801140    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801143    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801145    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801148    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801150    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801153    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801156    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801159    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801161    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801164    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801166    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:49.803980 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801169    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801172    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801174    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801177    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801180    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801182    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801185    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801188    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801190    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801193    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801196    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801198    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801201    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801204    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801206    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801209    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801212    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801214    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801217    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801220    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:49.804489 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801222    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801225    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801227    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801230    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801233    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801235    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801238    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801240    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801243    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801245    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801248    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801250    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801253    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801255    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801258    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801260    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801263    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801266    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801268    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801271    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:49.804985 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801273    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801276    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801279    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801283    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801286    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801289    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801292    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801295    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801299    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801302    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801305    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801307    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801310    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801313    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801315    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801318    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801321    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801324    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801326    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:49.805461 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801329    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801331    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801334    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801336    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801339    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801343    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801346    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:49.801349    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.801354    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.801453    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.803838    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.804913    2578 server.go:1019] "Starting client certificate rotation"
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.805014    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 18:50:49.805936 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.805047    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 18:50:49.807808 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.807781    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 18:50:49.811202 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.811182    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 18:50:49.818787 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.818774    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 15 18:50:49.822106 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.822092    2578 log.go:25] "Validated CRI v1 image API"
Sep 15 18:50:49.823778 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.823762    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 18:50:49.826488 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.826466    2578 fs.go:135] Filesystem UUIDs: map[551b7f8b-ff65-48eb-8f72-e5421c20c7af:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cad45b37-143d-409f-8c82-61a16449effe:/dev/nvme0n1p4]
Sep 15 18:50:49.826571 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.826486    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 18:50:49.831532 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.831513    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 18:50:49.833133 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.833032    2578 manager.go:216] Machine: {Timestamp:2026-09-15 18:50:49.831539157 +0000 UTC m=+0.373634678 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098142 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bea78734b559cf6521109baeec840 SystemUUID:ec2bea78-734b-559c-f652-1109baeec840 BootID:bfdff11e-f1b2-492b-be7b-7229670cf019 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:b1:c8:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:b1:c8:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:30:dd:dc:16:22 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 18:50:49.833133 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.833131    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 18:50:49.833231 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.833202    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 18:50:49.836828 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.836807    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 18:50:49.836955 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.836831    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-221.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 18:50:49.838436 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.838426    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 18:50:49.838468 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.838438    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 18:50:49.838468 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.838446    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 18:50:49.839160 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.839150    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 18:50:49.840374 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.840365    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 18:50:49.845066 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.845054    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 18:50:49.845165 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.845156    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 18:50:49.845210 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.845185    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 18:50:49.845210 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.845194    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 18:50:49.845210 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.845204    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 18:50:49.845322 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.845212    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 15 18:50:49.845322 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.845220    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 18:50:49.845513 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.845503    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 18:50:49.845557 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.845521    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 18:50:49.847875 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.847848    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 18:50:49.849913 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.849896    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nqsbx"
Sep 15 18:50:49.850754 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.850741    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 18:50:49.850812 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.850773    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 18:50:49.852889 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852875    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 18:50:49.852889 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852892    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 18:50:49.852990 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852901    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 18:50:49.852990 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852911    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 18:50:49.852990 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852921    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 18:50:49.852990 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852933    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 18:50:49.852990 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852946    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 18:50:49.852990 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852951    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 18:50:49.852990 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852958    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 18:50:49.852990 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852963    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 18:50:49.852990 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852982    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 18:50:49.852990 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.852987    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 18:50:49.853858 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.853844    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 18:50:49.853858 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.853858    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 18:50:49.854482 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:49.854462    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 18:50:49.854525 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:49.854451    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 18:50:49.856762 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.856742    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nqsbx"
Sep 15 18:50:49.858242 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.858222    2578 server.go:1268] "Started kubelet"
Sep 15 18:50:49.858336 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.858281    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 18:50:49.858434 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.858341    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 18:50:49.858475 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.858450    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 18:50:49.859000 ip-10-0-135-221 systemd[1]: Started Kubernetes Kubelet.
Sep 15 18:50:49.860055 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.860039    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 18:50:49.861461 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.861096    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 18:50:49.863907 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:49.863876    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 18:50:49.869194 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.869170    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 18:50:49.869856 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.869839    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 18:50:49.870064 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:49.870019    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:49.870551 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.870527    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 18:50:49.870551 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.870531    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 18:50:49.870678 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.870559    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 18:50:49.870723 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.870700    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 18:50:49.870723 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.870710    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 18:50:49.870906 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.870890    2578 factory.go:55] Registering systemd factory
Sep 15 18:50:49.870952 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.870909    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 15 18:50:49.871145 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.871132    2578 factory.go:153] Registering CRI-O factory
Sep 15 18:50:49.871190 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.871149    2578 factory.go:223] Registration of the crio container factory successfully
Sep 15 18:50:49.871238 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.871191    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 18:50:49.871238 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.871230    2578 factory.go:103] Registering Raw factory
Sep 15 18:50:49.871313 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.871253    2578 manager.go:1196] Started watching for new ooms in manager
Sep 15 18:50:49.871572 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.871557    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-221.ec2.internal" not found
Sep 15 18:50:49.871773 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.871760    2578 manager.go:349] Starting recovery of all containers
Sep 15 18:50:49.872835 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.872783    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:49.877280 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:49.877258    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-221.ec2.internal\" not found" node="ip-10-0-135-221.ec2.internal"
Sep 15 18:50:49.886664 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.886646    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-221.ec2.internal" not found
Sep 15 18:50:49.887325 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.887312    2578 manager.go:354] Recovery completed
Sep 15 18:50:49.891449 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.891359    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:49.893122 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.893105    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:49.893176 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.893136    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:49.893176 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.893149    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:49.893623 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.893610    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:49.893732 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.893611    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 18:50:49.893732 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.893647    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 18:50:49.893732 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.893676    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 18:50:49.894254 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.894237    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:49.894328 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.894264    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:49.894328 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.894277    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:49.895903 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.895889    2578 policy_none.go:49] "None policy: Start"
Sep 15 18:50:49.895903 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.895906    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 18:50:49.896000 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.895916    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 18:50:49.897970 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.897958    2578 policy_none.go:47] "Start"
Sep 15 18:50:49.943346 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.943324    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-221.ec2.internal" not found
Sep 15 18:50:49.943432 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.943381    2578 manager.go:337] "Starting Device Plugin manager"
Sep 15 18:50:49.943432 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:49.943416    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 18:50:49.943432 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.943429    2578 server.go:87] "Starting device plugin registration server"
Sep 15 18:50:49.943706 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.943693    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 18:50:49.943765 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.943708    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 18:50:49.943830 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.943817    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 18:50:49.943905 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.943889    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 18:50:49.943905 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.943902    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 18:50:49.944424 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:49.944406    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 18:50:49.944507 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:49.944455    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:49.957965 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:49.957948    2578 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2606 into the system container \"/system.slice\": <nil>"
Sep 15 18:50:50.044616 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.044558    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:50.045294 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.045279    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:50.045362 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.045309    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:50.045362 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.045324    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:50.045362 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.045355    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.055232 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.055216    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.070209 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.055235    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-221.ec2.internal\": node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:50.070209 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.069324    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:50.091147 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.091111    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 18:50:50.092259 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.092247    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 18:50:50.092323 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.092291    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 18:50:50.092323 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.092316    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 18:50:50.092399 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.092387    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 18:50:50.094735 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.094716    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:50.169842 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.169826    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:50.193165 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.193149    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal"]
Sep 15 18:50:50.193226 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.193210    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:50.193958 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.193945    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:50.194026 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.193971    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:50.194026 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.193983    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:50.196229 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.196216    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:50.196385 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.196370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.196446 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.196406    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:50.196989 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.196971    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:50.197075 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.196973    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:50.197075 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.197022    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:50.197075 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.197033    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:50.197075 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.197002    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:50.197207 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.197077    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:50.199204 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.199189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.199295 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.199215    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:50.199906 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.199890    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:50.199998 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.199914    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:50.199998 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.199928    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:50.226424 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.226401    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-221.ec2.internal\" not found" node="ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.230934 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.230919    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-221.ec2.internal\" not found" node="ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.270013 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.269993    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:50.273082 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.273067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f58418f2eb3a96678dd7e1f043ca38f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-221.ec2.internal\" (UID: \"1f58418f2eb3a96678dd7e1f043ca38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.273158 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.273089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89c1bc29e834f6f6903ce15b6a1f531c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal\" (UID: \"89c1bc29e834f6f6903ce15b6a1f531c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.273158 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.273109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89c1bc29e834f6f6903ce15b6a1f531c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal\" (UID: \"89c1bc29e834f6f6903ce15b6a1f531c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.370370 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.370316    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:50.373567 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.373553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f58418f2eb3a96678dd7e1f043ca38f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-221.ec2.internal\" (UID: \"1f58418f2eb3a96678dd7e1f043ca38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.373623 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.373574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89c1bc29e834f6f6903ce15b6a1f531c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal\" (UID: \"89c1bc29e834f6f6903ce15b6a1f531c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.373623 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.373589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89c1bc29e834f6f6903ce15b6a1f531c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal\" (UID: \"89c1bc29e834f6f6903ce15b6a1f531c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.373698 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.373647    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89c1bc29e834f6f6903ce15b6a1f531c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal\" (UID: \"89c1bc29e834f6f6903ce15b6a1f531c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.373698 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.373665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f58418f2eb3a96678dd7e1f043ca38f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-221.ec2.internal\" (UID: \"1f58418f2eb3a96678dd7e1f043ca38f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.373768 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.373665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89c1bc29e834f6f6903ce15b6a1f531c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal\" (UID: \"89c1bc29e834f6f6903ce15b6a1f531c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.470752 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.470715    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:50.529038 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.529018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.533895 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.533861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.571588 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.571562    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:50.672009 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.671961    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:50.772453 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.772433    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:50.804731 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.804713    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 18:50:50.805152 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.804871    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:50.805152 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.804878    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:50.864839 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.864772    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 18:45:50 +0000 UTC" deadline="2028-05-26 10:32:08.069908726 +0000 UTC"
Sep 15 18:50:50.864839 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.864833    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14847h41m17.205078522s"
Sep 15 18:50:50.872717 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:50.872695    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-221.ec2.internal\" not found"
Sep 15 18:50:50.914965 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.914943    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:50.971476 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.971432    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.983225 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.983204    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 18:50:50.984727 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.984714    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal"
Sep 15 18:50:50.991221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:50.991205    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 18:50:51.141434 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.141403    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:51.261088 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:51.261052    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f58418f2eb3a96678dd7e1f043ca38f.slice/crio-a5c161c92aa5819f5ee6cd89a43b0a79199bd7989cbe5e616ae429e88e460786 WatchSource:0}: Error finding container a5c161c92aa5819f5ee6cd89a43b0a79199bd7989cbe5e616ae429e88e460786: Status 404 returned error can't find the container with id a5c161c92aa5819f5ee6cd89a43b0a79199bd7989cbe5e616ae429e88e460786
Sep 15 18:50:51.261581 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:51.261552    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c1bc29e834f6f6903ce15b6a1f531c.slice/crio-289bbb49abae68f443f980cb1b9ff8b76d3163afc6fce7410f78347e1c9194a2 WatchSource:0}: Error finding container 289bbb49abae68f443f980cb1b9ff8b76d3163afc6fce7410f78347e1c9194a2: Status 404 returned error can't find the container with id 289bbb49abae68f443f980cb1b9ff8b76d3163afc6fce7410f78347e1c9194a2
Sep 15 18:50:51.265953 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.265934    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 18:50:51.710273 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.710191    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:51.846721 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.846680    2578 apiserver.go:52] "Watching apiserver"
Sep 15 18:50:51.856885 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.856860    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 18:50:51.857281 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.857258    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-srpw5","kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7d7qs","openshift-dns/node-resolver-4gljt","openshift-multus/multus-9nr27","openshift-multus/multus-additional-cni-plugins-b75vj","openshift-network-diagnostics/network-check-target-k8tcg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8","openshift-image-registry/node-ca-4glsv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal","openshift-multus/network-metrics-daemon-hktjq","openshift-network-operator/iptables-alerter-6p5c5","openshift-ovn-kubernetes/ovnkube-node-qg5zh","kube-system/global-pull-secret-syncer-9qhdm"]
Sep 15 18:50:51.859889 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.859866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:51.860004 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:51.859958    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:50:51.862517 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.862497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.864741 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.864707    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 18:50:51.865442 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.864906    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 18:50:51.865442 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.864958    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 18:50:51.865442 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.864978    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 18:50:51.865442 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.865150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.865442 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.865270    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-btcwk\""
Sep 15 18:50:51.865923 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.865906    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 18:50:51.865923 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.865936    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 18:50:51.867628 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.867444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:51.868094 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.868069    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5rn97\""
Sep 15 18:50:51.868243 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.868216    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 18:50:51.868346 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.868289    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:50:51.869928 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.869909    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 18:50:51.870164 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.870149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.870387 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.870368    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlffw\""
Sep 15 18:50:51.870467 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.870399    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 18:50:51.871475 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.871453    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 18:50:51.873013 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.872955    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 18:50:51.873108 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.873062    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rklgq\""
Sep 15 18:50:51.873621 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.873604    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 18:50:51.874670 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.873998    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 18:50:51.874670 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.874149    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 18:50:51.874670 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.874180    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.876437 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.876416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:51.877079 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.877060    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 18:50:51.877280 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.877066    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 18:50:51.877355 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.877083    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t26b7\""
Sep 15 18:50:51.878353 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.878335    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8tmqk\""
Sep 15 18:50:51.878548 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.878526    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 18:50:51.878722 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.878700    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 18:50:51.878793 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.878740    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 18:50:51.878999 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.878983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:50:51.881039 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.880994    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 18:50:51.881256 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.881239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2m2lq\""
Sep 15 18:50:51.881410 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.881388    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 18:50:51.881631 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.881611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:51.881711 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:51.881693    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:50:51.882317 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-etc-kubernetes\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.882417 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-kubelet\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.882417 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2845410e-10e0-4248-862b-1d7bed1fd00d-cni-binary-copy\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.882417 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30cfe649-c99b-45ad-9325-c187e24073c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.882536 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-systemd\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.882536 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-run-k8s-cni-cncf-io\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.882536 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-var-lib-openvswitch\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.882536 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-host\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.882733 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrf5k\" (UniqueName: \"kubernetes.io/projected/9e6b9574-0dfc-4290-b6ab-a785808d21a4-kube-api-access-wrf5k\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.882733 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-conf-dir\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.882733 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-run-netns\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.882733 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-run-openvswitch\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.882733 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-modprobe-d\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.882733 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-var-lib-cni-multus\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.882733 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-hostroot\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.883070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-daemon-config\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.883070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-slash\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-run-systemd\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06644277-8f48-4af9-930e-c6493a57b420-hosts-file\") pod \"node-resolver-4gljt\" (UID: \"06644277-8f48-4af9-930e-c6493a57b420\") " pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:51.883070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-etc-openvswitch\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96e24b14-fd77-449d-9ee6-6b7bb141a419-ovnkube-config\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.882982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96e24b14-fd77-449d-9ee6-6b7bb141a419-env-overrides\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-sysconfig\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.883070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-tuned\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.883070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-run-netns\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-var-lib-cni-bin\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-run-ovn\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-log-socket\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mww8c\" (UniqueName: \"kubernetes.io/projected/96e24b14-fd77-449d-9ee6-6b7bb141a419-kube-api-access-mww8c\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-system-cni-dir\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-run-ovn-kubernetes\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-node-log\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-cni-bin\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-sysctl-d\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-var-lib-kubelet\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30cfe649-c99b-45ad-9325-c187e24073c3-cnibin\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96e24b14-fd77-449d-9ee6-6b7bb141a419-ovn-node-metrics-cert\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.883493 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-lib-modules\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6b9574-0dfc-4290-b6ab-a785808d21a4-tmp\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22r7s\" (UniqueName: \"kubernetes.io/projected/2845410e-10e0-4248-862b-1d7bed1fd00d-kube-api-access-22r7s\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-systemd-units\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-kubernetes\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-sys\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-cni-dir\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-run-multus-certs\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30cfe649-c99b-45ad-9325-c187e24073c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6xt\" (UniqueName: \"kubernetes.io/projected/30cfe649-c99b-45ad-9325-c187e24073c3-kube-api-access-xc6xt\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbklb\" (UniqueName: \"kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb\") pod \"network-check-target-k8tcg\" (UID: \"b968fefe-3155-4c6c-8aee-c00f3af85ad0\") " pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96e24b14-fd77-449d-9ee6-6b7bb141a419-ovnkube-script-lib\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-cnibin\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-var-lib-kubelet\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.883951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30cfe649-c99b-45ad-9325-c187e24073c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.884000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-sysctl-conf\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.884189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.884037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06644277-8f48-4af9-930e-c6493a57b420-tmp-dir\") pod \"node-resolver-4gljt\" (UID: \"06644277-8f48-4af9-930e-c6493a57b420\") " pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:51.884906 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.884078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9n7w\" (UniqueName: \"kubernetes.io/projected/06644277-8f48-4af9-930e-c6493a57b420-kube-api-access-x9n7w\") pod \"node-resolver-4gljt\" (UID: \"06644277-8f48-4af9-930e-c6493a57b420\") " pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:51.884906 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.884105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30cfe649-c99b-45ad-9325-c187e24073c3-system-cni-dir\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.884906 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.884141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30cfe649-c99b-45ad-9325-c187e24073c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.884906 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.884170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-cni-netd\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.884906 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.884209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-run\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.884906 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.884240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-socket-dir-parent\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.884906 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.884319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:51.886768 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.886635    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 18:50:51.886899 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.886800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 18:50:51.886960 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.886919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:51.887022 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:51.886983    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:50:51.887315 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.887216    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 18:50:51.887315 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.887220    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9qv9r\""
Sep 15 18:50:51.887473 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.887405    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 18:50:51.889527 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.889506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:51.891758 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.891734    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:50:51.891879 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.891739    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 18:50:51.891959 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.891927    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8jbfc\""
Sep 15 18:50:51.892032 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.891956    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 18:50:51.913199 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.913177    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzvfg"
Sep 15 18:50:51.921554 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.921535    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzvfg"
Sep 15 18:50:51.971517 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.971459    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 18:50:51.984545 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-run-systemd\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.984642 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06644277-8f48-4af9-930e-c6493a57b420-hosts-file\") pod \"node-resolver-4gljt\" (UID: \"06644277-8f48-4af9-930e-c6493a57b420\") " pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:51.984642 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:51.984756 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984651    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-run-systemd\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.984756 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06644277-8f48-4af9-930e-c6493a57b420-hosts-file\") pod \"node-resolver-4gljt\" (UID: \"06644277-8f48-4af9-930e-c6493a57b420\") " pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:51.984756 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-etc-openvswitch\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.984907 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96e24b14-fd77-449d-9ee6-6b7bb141a419-ovnkube-config\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.984907 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-etc-openvswitch\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.984907 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96e24b14-fd77-449d-9ee6-6b7bb141a419-env-overrides\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.984907 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-sysconfig\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.984907 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-tuned\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.984907 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-run-netns\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-var-lib-cni-bin\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-run-ovn\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-log-socket\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-run-netns\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.984997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mww8c\" (UniqueName: \"kubernetes.io/projected/96e24b14-fd77-449d-9ee6-6b7bb141a419-kube-api-access-mww8c\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac29fcda-9928-4ad9-9cee-6456cabf59a0-serviceca\") pod \"node-ca-4glsv\" (UID: \"ac29fcda-9928-4ad9-9cee-6456cabf59a0\") " pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-system-cni-dir\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985049    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-var-lib-cni-bin\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-run-ovn-kubernetes\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-sysconfig\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-socket-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klfn2\" (UniqueName: \"kubernetes.io/projected/e94d481d-d5f0-481d-9b9b-b3924ae440fe-kube-api-access-klfn2\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-log-socket\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-node-log\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.985221 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-cni-bin\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-sysctl-d\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-node-log\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-run-ovn-kubernetes\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985316    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96e24b14-fd77-449d-9ee6-6b7bb141a419-env-overrides\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-system-cni-dir\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-var-lib-kubelet\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-run-ovn\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3ef2522-f438-42f8-a961-b5b62e5c4836-dbus\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96e24b14-fd77-449d-9ee6-6b7bb141a419-ovnkube-config\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-cni-bin\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-var-lib-kubelet\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtl28\" (UniqueName: \"kubernetes.io/projected/eb8d5cee-7842-4f44-9079-94434a03ccff-kube-api-access-xtl28\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-sysctl-d\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30cfe649-c99b-45ad-9325-c187e24073c3-cnibin\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.986050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30cfe649-c99b-45ad-9325-c187e24073c3-cnibin\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96e24b14-fd77-449d-9ee6-6b7bb141a419-ovn-node-metrics-cert\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-lib-modules\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6b9574-0dfc-4290-b6ab-a785808d21a4-tmp\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac29fcda-9928-4ad9-9cee-6456cabf59a0-host\") pod \"node-ca-4glsv\" (UID: \"ac29fcda-9928-4ad9-9cee-6456cabf59a0\") " pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-registration-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22r7s\" (UniqueName: \"kubernetes.io/projected/2845410e-10e0-4248-862b-1d7bed1fd00d-kube-api-access-22r7s\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-lib-modules\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-systemd-units\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-kubernetes\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-sys\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-systemd-units\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b251128d-040c-49f9-8396-36a2db07a539-agent-certs\") pod \"konnectivity-agent-srpw5\" (UID: \"b251128d-040c-49f9-8396-36a2db07a539\") " pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-cni-dir\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-run-multus-certs\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30cfe649-c99b-45ad-9325-c187e24073c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.987023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6xt\" (UniqueName: \"kubernetes.io/projected/30cfe649-c99b-45ad-9325-c187e24073c3-kube-api-access-xc6xt\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbklb\" (UniqueName: \"kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb\") pod \"network-check-target-k8tcg\" (UID: \"b968fefe-3155-4c6c-8aee-c00f3af85ad0\") " pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-kubernetes\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-sys-fs\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96e24b14-fd77-449d-9ee6-6b7bb141a419-ovnkube-script-lib\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-sys\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.985996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-cni-dir\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-run-multus-certs\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npgtg\" (UniqueName: \"kubernetes.io/projected/ac29fcda-9928-4ad9-9cee-6456cabf59a0-kube-api-access-npgtg\") pod \"node-ca-4glsv\" (UID: \"ac29fcda-9928-4ad9-9cee-6456cabf59a0\") " pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-cnibin\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-var-lib-kubelet\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30cfe649-c99b-45ad-9325-c187e24073c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-sysctl-conf\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06644277-8f48-4af9-930e-c6493a57b420-tmp-dir\") pod \"node-resolver-4gljt\" (UID: \"06644277-8f48-4af9-930e-c6493a57b420\") " pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9n7w\" (UniqueName: \"kubernetes.io/projected/06644277-8f48-4af9-930e-c6493a57b420-kube-api-access-x9n7w\") pod \"node-resolver-4gljt\" (UID: \"06644277-8f48-4af9-930e-c6493a57b420\") " pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-var-lib-kubelet\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30cfe649-c99b-45ad-9325-c187e24073c3-system-cni-dir\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.987844 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30cfe649-c99b-45ad-9325-c187e24073c3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30cfe649-c99b-45ad-9325-c187e24073c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-cni-netd\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-run\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99b30109-dc43-48f1-a6d9-999575a5b20b-iptables-alerter-script\") pod \"iptables-alerter-6p5c5\" (UID: \"99b30109-dc43-48f1-a6d9-999575a5b20b\") " pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-socket-dir-parent\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-run\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-etc-kubernetes\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-kubelet\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30cfe649-c99b-45ad-9325-c187e24073c3-system-cni-dir\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cvv\" (UniqueName: \"kubernetes.io/projected/99b30109-dc43-48f1-a6d9-999575a5b20b-kube-api-access-69cvv\") pod \"iptables-alerter-6p5c5\" (UID: \"99b30109-dc43-48f1-a6d9-999575a5b20b\") " pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30cfe649-c99b-45ad-9325-c187e24073c3-cni-binary-copy\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2845410e-10e0-4248-862b-1d7bed1fd00d-cni-binary-copy\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30cfe649-c99b-45ad-9325-c187e24073c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-cni-netd\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-systemd\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.988791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-run-k8s-cni-cncf-io\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-var-lib-openvswitch\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-host\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06644277-8f48-4af9-930e-c6493a57b420-tmp-dir\") pod \"node-resolver-4gljt\" (UID: \"06644277-8f48-4af9-930e-c6493a57b420\") " pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrf5k\" (UniqueName: \"kubernetes.io/projected/9e6b9574-0dfc-4290-b6ab-a785808d21a4-kube-api-access-wrf5k\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-sysctl-conf\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96e24b14-fd77-449d-9ee6-6b7bb141a419-ovnkube-script-lib\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.986996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99b30109-dc43-48f1-a6d9-999575a5b20b-host-slash\") pod \"iptables-alerter-6p5c5\" (UID: \"99b30109-dc43-48f1-a6d9-999575a5b20b\") " pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.987413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30cfe649-c99b-45ad-9325-c187e24073c3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.987444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-run-k8s-cni-cncf-io\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.987506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-cnibin\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.987558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-systemd\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.987586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-socket-dir-parent\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.987637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-etc-kubernetes\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.987691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-var-lib-openvswitch\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.987697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30cfe649-c99b-45ad-9325-c187e24073c3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.987930    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-kubelet\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.987969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-conf-dir\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.989519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-run-netns\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-run-openvswitch\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-modprobe-d\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3ef2522-f438-42f8-a961-b5b62e5c4836-kubelet-config\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b251128d-040c-49f9-8396-36a2db07a539-konnectivity-ca\") pod \"konnectivity-agent-srpw5\" (UID: \"b251128d-040c-49f9-8396-36a2db07a539\") " pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-device-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-var-lib-cni-multus\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-hostroot\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-daemon-config\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2845410e-10e0-4248-862b-1d7bed1fd00d-cni-binary-copy\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-slash\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-host\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-host-var-lib-cni-multus\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-conf-dir\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-run-netns\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-run-openvswitch\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988455    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-modprobe-d\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2845410e-10e0-4248-862b-1d7bed1fd00d-hostroot\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.990277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96e24b14-fd77-449d-9ee6-6b7bb141a419-host-slash\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.991076 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.988811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2845410e-10e0-4248-862b-1d7bed1fd00d-multus-daemon-config\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.991076 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.989612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6b9574-0dfc-4290-b6ab-a785808d21a4-tmp\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.991076 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.989638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9e6b9574-0dfc-4290-b6ab-a785808d21a4-etc-tuned\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.991258 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.991236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96e24b14-fd77-449d-9ee6-6b7bb141a419-ovn-node-metrics-cert\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:51.992746 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:51.992722    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:51.992873 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:51.992752    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:51.992873 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:51.992766    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nbklb for pod openshift-network-diagnostics/network-check-target-k8tcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:51.992993 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:51.992872    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb podName:b968fefe-3155-4c6c-8aee-c00f3af85ad0 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:52.492849742 +0000 UTC m=+3.034945265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nbklb" (UniqueName: "kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb") pod "network-check-target-k8tcg" (UID: "b968fefe-3155-4c6c-8aee-c00f3af85ad0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:51.994239 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.994216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22r7s\" (UniqueName: \"kubernetes.io/projected/2845410e-10e0-4248-862b-1d7bed1fd00d-kube-api-access-22r7s\") pod \"multus-9nr27\" (UID: \"2845410e-10e0-4248-862b-1d7bed1fd00d\") " pod="openshift-multus/multus-9nr27"
Sep 15 18:50:51.994717 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.994626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9n7w\" (UniqueName: \"kubernetes.io/projected/06644277-8f48-4af9-930e-c6493a57b420-kube-api-access-x9n7w\") pod \"node-resolver-4gljt\" (UID: \"06644277-8f48-4af9-930e-c6493a57b420\") " pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:51.994919 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.994897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrf5k\" (UniqueName: \"kubernetes.io/projected/9e6b9574-0dfc-4290-b6ab-a785808d21a4-kube-api-access-wrf5k\") pod \"tuned-7d7qs\" (UID: \"9e6b9574-0dfc-4290-b6ab-a785808d21a4\") " pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:51.994979 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.994863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6xt\" (UniqueName: \"kubernetes.io/projected/30cfe649-c99b-45ad-9325-c187e24073c3-kube-api-access-xc6xt\") pod \"multus-additional-cni-plugins-b75vj\" (UID: \"30cfe649-c99b-45ad-9325-c187e24073c3\") " pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:51.995338 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:51.995320    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mww8c\" (UniqueName: \"kubernetes.io/projected/96e24b14-fd77-449d-9ee6-6b7bb141a419-kube-api-access-mww8c\") pod \"ovnkube-node-qg5zh\" (UID: \"96e24b14-fd77-449d-9ee6-6b7bb141a419\") " pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:52.089275 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99b30109-dc43-48f1-a6d9-999575a5b20b-iptables-alerter-script\") pod \"iptables-alerter-6p5c5\" (UID: \"99b30109-dc43-48f1-a6d9-999575a5b20b\") " pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:52.089275 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:52.089489 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cvv\" (UniqueName: \"kubernetes.io/projected/99b30109-dc43-48f1-a6d9-999575a5b20b-kube-api-access-69cvv\") pod \"iptables-alerter-6p5c5\" (UID: \"99b30109-dc43-48f1-a6d9-999575a5b20b\") " pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:52.089489 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99b30109-dc43-48f1-a6d9-999575a5b20b-host-slash\") pod \"iptables-alerter-6p5c5\" (UID: \"99b30109-dc43-48f1-a6d9-999575a5b20b\") " pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:52.089489 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3ef2522-f438-42f8-a961-b5b62e5c4836-kubelet-config\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:52.089489 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b251128d-040c-49f9-8396-36a2db07a539-konnectivity-ca\") pod \"konnectivity-agent-srpw5\" (UID: \"b251128d-040c-49f9-8396-36a2db07a539\") " pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:50:52.089489 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.089396    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:52.089489 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089454    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-device-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.089489 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.089475    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs podName:eb8d5cee-7842-4f44-9079-94434a03ccff nodeName:}" failed. No retries permitted until 2026-09-15 18:50:52.589454389 +0000 UTC m=+3.131549920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs") pod "network-metrics-daemon-hktjq" (UID: "eb8d5cee-7842-4f44-9079-94434a03ccff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:52.089489 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-device-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089500    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99b30109-dc43-48f1-a6d9-999575a5b20b-host-slash\") pod \"iptables-alerter-6p5c5\" (UID: \"99b30109-dc43-48f1-a6d9-999575a5b20b\") " pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3ef2522-f438-42f8-a961-b5b62e5c4836-kubelet-config\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac29fcda-9928-4ad9-9cee-6456cabf59a0-serviceca\") pod \"node-ca-4glsv\" (UID: \"ac29fcda-9928-4ad9-9cee-6456cabf59a0\") " pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-socket-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klfn2\" (UniqueName: \"kubernetes.io/projected/e94d481d-d5f0-481d-9b9b-b3924ae440fe-kube-api-access-klfn2\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3ef2522-f438-42f8-a961-b5b62e5c4836-dbus\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtl28\" (UniqueName: \"kubernetes.io/projected/eb8d5cee-7842-4f44-9079-94434a03ccff-kube-api-access-xtl28\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac29fcda-9928-4ad9-9cee-6456cabf59a0-host\") pod \"node-ca-4glsv\" (UID: \"ac29fcda-9928-4ad9-9cee-6456cabf59a0\") " pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-registration-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b251128d-040c-49f9-8396-36a2db07a539-agent-certs\") pod \"konnectivity-agent-srpw5\" (UID: \"b251128d-040c-49f9-8396-36a2db07a539\") " pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-socket-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-sys-fs\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.089934 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npgtg\" (UniqueName: \"kubernetes.io/projected/ac29fcda-9928-4ad9-9cee-6456cabf59a0-kube-api-access-npgtg\") pod \"node-ca-4glsv\" (UID: \"ac29fcda-9928-4ad9-9cee-6456cabf59a0\") " pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:52.090575 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.089923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.090575 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.090026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99b30109-dc43-48f1-a6d9-999575a5b20b-iptables-alerter-script\") pod \"iptables-alerter-6p5c5\" (UID: \"99b30109-dc43-48f1-a6d9-999575a5b20b\") " pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:52.090575 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.090062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac29fcda-9928-4ad9-9cee-6456cabf59a0-serviceca\") pod \"node-ca-4glsv\" (UID: \"ac29fcda-9928-4ad9-9cee-6456cabf59a0\") " pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:52.090575 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.090132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-registration-dir\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.090575 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.090145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b251128d-040c-49f9-8396-36a2db07a539-konnectivity-ca\") pod \"konnectivity-agent-srpw5\" (UID: \"b251128d-040c-49f9-8396-36a2db07a539\") " pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:50:52.090575 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.090174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac29fcda-9928-4ad9-9cee-6456cabf59a0-host\") pod \"node-ca-4glsv\" (UID: \"ac29fcda-9928-4ad9-9cee-6456cabf59a0\") " pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:52.090575 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.090210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3ef2522-f438-42f8-a961-b5b62e5c4836-dbus\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:52.090575 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.090255    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:52.090575 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.090259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e94d481d-d5f0-481d-9b9b-b3924ae440fe-sys-fs\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.090575 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.090318    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret podName:e3ef2522-f438-42f8-a961-b5b62e5c4836 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:52.590301058 +0000 UTC m=+3.132396588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret") pod "global-pull-secret-syncer-9qhdm" (UID: "e3ef2522-f438-42f8-a961-b5b62e5c4836") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:52.092826 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.092789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b251128d-040c-49f9-8396-36a2db07a539-agent-certs\") pod \"konnectivity-agent-srpw5\" (UID: \"b251128d-040c-49f9-8396-36a2db07a539\") " pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:50:52.097292 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.097253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal" event={"ID":"89c1bc29e834f6f6903ce15b6a1f531c","Type":"ContainerStarted","Data":"289bbb49abae68f443f980cb1b9ff8b76d3163afc6fce7410f78347e1c9194a2"}
Sep 15 18:50:52.098850 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.098798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal" event={"ID":"1f58418f2eb3a96678dd7e1f043ca38f","Type":"ContainerStarted","Data":"a5c161c92aa5819f5ee6cd89a43b0a79199bd7989cbe5e616ae429e88e460786"}
Sep 15 18:50:52.101968 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.101934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cvv\" (UniqueName: \"kubernetes.io/projected/99b30109-dc43-48f1-a6d9-999575a5b20b-kube-api-access-69cvv\") pod \"iptables-alerter-6p5c5\" (UID: \"99b30109-dc43-48f1-a6d9-999575a5b20b\") " pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:52.102238 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.102216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtl28\" (UniqueName: \"kubernetes.io/projected/eb8d5cee-7842-4f44-9079-94434a03ccff-kube-api-access-xtl28\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:52.102561 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.102533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klfn2\" (UniqueName: \"kubernetes.io/projected/e94d481d-d5f0-481d-9b9b-b3924ae440fe-kube-api-access-klfn2\") pod \"aws-ebs-csi-driver-node-pnrq8\" (UID: \"e94d481d-d5f0-481d-9b9b-b3924ae440fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.102633 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.102598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npgtg\" (UniqueName: \"kubernetes.io/projected/ac29fcda-9928-4ad9-9cee-6456cabf59a0-kube-api-access-npgtg\") pod \"node-ca-4glsv\" (UID: \"ac29fcda-9928-4ad9-9cee-6456cabf59a0\") " pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:52.176853 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.176832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:50:52.184994 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.184968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7d7qs"
Sep 15 18:50:52.189118 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:52.189090    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e24b14_fd77_449d_9ee6_6b7bb141a419.slice/crio-793ab2f0b515033ce10a6e4108410a62ecae47593b705cca14f269231cd9458c WatchSource:0}: Error finding container 793ab2f0b515033ce10a6e4108410a62ecae47593b705cca14f269231cd9458c: Status 404 returned error can't find the container with id 793ab2f0b515033ce10a6e4108410a62ecae47593b705cca14f269231cd9458c
Sep 15 18:50:52.195149 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:52.195126    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6b9574_0dfc_4290_b6ab_a785808d21a4.slice/crio-127973cf790f87fa2106b27f391fdbe8ccca66efffa857b94959200b1940d65b WatchSource:0}: Error finding container 127973cf790f87fa2106b27f391fdbe8ccca66efffa857b94959200b1940d65b: Status 404 returned error can't find the container with id 127973cf790f87fa2106b27f391fdbe8ccca66efffa857b94959200b1940d65b
Sep 15 18:50:52.196484 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.196458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4glsv"
Sep 15 18:50:52.202208 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.202179    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4gljt"
Sep 15 18:50:52.207228 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:52.207204    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac29fcda_9928_4ad9_9cee_6456cabf59a0.slice/crio-4f1c102997173ae353a6f0ae459fcb5031d3e08f6930e93b72382f7b3e459c16 WatchSource:0}: Error finding container 4f1c102997173ae353a6f0ae459fcb5031d3e08f6930e93b72382f7b3e459c16: Status 404 returned error can't find the container with id 4f1c102997173ae353a6f0ae459fcb5031d3e08f6930e93b72382f7b3e459c16
Sep 15 18:50:52.210919 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.210897    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9nr27"
Sep 15 18:50:52.212836 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:52.212799    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06644277_8f48_4af9_930e_c6493a57b420.slice/crio-9a82a60b005410ed0da25eeaf8c0220e6251ec666a061afe09d1a892efa4f236 WatchSource:0}: Error finding container 9a82a60b005410ed0da25eeaf8c0220e6251ec666a061afe09d1a892efa4f236: Status 404 returned error can't find the container with id 9a82a60b005410ed0da25eeaf8c0220e6251ec666a061afe09d1a892efa4f236
Sep 15 18:50:52.218911 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.218887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b75vj"
Sep 15 18:50:52.221178 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:52.221151    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2845410e_10e0_4248_862b_1d7bed1fd00d.slice/crio-55cab18ba12d7f7faf8bb54a9c1cf64fe1f1e1869137d373bf8960b7065b43c0 WatchSource:0}: Error finding container 55cab18ba12d7f7faf8bb54a9c1cf64fe1f1e1869137d373bf8960b7065b43c0: Status 404 returned error can't find the container with id 55cab18ba12d7f7faf8bb54a9c1cf64fe1f1e1869137d373bf8960b7065b43c0
Sep 15 18:50:52.226255 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.226227    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:50:52.231026 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:52.230999    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cfe649_c99b_45ad_9325_c187e24073c3.slice/crio-24b72f8716a31ffd12cb640de4fc7b3638716228775d69ed79bb4a6c94878e17 WatchSource:0}: Error finding container 24b72f8716a31ffd12cb640de4fc7b3638716228775d69ed79bb4a6c94878e17: Status 404 returned error can't find the container with id 24b72f8716a31ffd12cb640de4fc7b3638716228775d69ed79bb4a6c94878e17
Sep 15 18:50:52.233303 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.233145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8"
Sep 15 18:50:52.236113 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:52.236093    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb251128d_040c_49f9_8396_36a2db07a539.slice/crio-f826c710a1e2cbf2cc4bcd4cd430f50e430516e6e11a392e160fd0522b823071 WatchSource:0}: Error finding container f826c710a1e2cbf2cc4bcd4cd430f50e430516e6e11a392e160fd0522b823071: Status 404 returned error can't find the container with id f826c710a1e2cbf2cc4bcd4cd430f50e430516e6e11a392e160fd0522b823071
Sep 15 18:50:52.239580 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.239559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6p5c5"
Sep 15 18:50:52.243794 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:52.243766    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d481d_d5f0_481d_9b9b_b3924ae440fe.slice/crio-e3a114669b56623c0968d5503eaea416d7eab28419616bff2d67e84277a60f88 WatchSource:0}: Error finding container e3a114669b56623c0968d5503eaea416d7eab28419616bff2d67e84277a60f88: Status 404 returned error can't find the container with id e3a114669b56623c0968d5503eaea416d7eab28419616bff2d67e84277a60f88
Sep 15 18:50:52.251030 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:50:52.251004    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b30109_dc43_48f1_a6d9_999575a5b20b.slice/crio-7a71df26516d5429660e98cc91c7e89f512a1d8320521ff0856f18af246cf7e0 WatchSource:0}: Error finding container 7a71df26516d5429660e98cc91c7e89f512a1d8320521ff0856f18af246cf7e0: Status 404 returned error can't find the container with id 7a71df26516d5429660e98cc91c7e89f512a1d8320521ff0856f18af246cf7e0
Sep 15 18:50:52.303820 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.303767    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:52.493207 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.493106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbklb\" (UniqueName: \"kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb\") pod \"network-check-target-k8tcg\" (UID: \"b968fefe-3155-4c6c-8aee-c00f3af85ad0\") " pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:52.493338 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.493254    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:52.493338 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.493272    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:52.493338 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.493282    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nbklb for pod openshift-network-diagnostics/network-check-target-k8tcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:52.493338 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.493335    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb podName:b968fefe-3155-4c6c-8aee-c00f3af85ad0 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:53.493321332 +0000 UTC m=+4.035416841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbklb" (UniqueName: "kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb") pod "network-check-target-k8tcg" (UID: "b968fefe-3155-4c6c-8aee-c00f3af85ad0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:52.593772 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.593737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:52.594004 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.593897    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:52.594004 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.593942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:52.594004 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.593984    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs podName:eb8d5cee-7842-4f44-9079-94434a03ccff nodeName:}" failed. No retries permitted until 2026-09-15 18:50:53.593958321 +0000 UTC m=+4.136053856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs") pod "network-metrics-daemon-hktjq" (UID: "eb8d5cee-7842-4f44-9079-94434a03ccff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:52.594188 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.594035    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:52.594188 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:52.594091    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret podName:e3ef2522-f438-42f8-a961-b5b62e5c4836 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:53.594073248 +0000 UTC m=+4.136168773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret") pod "global-pull-secret-syncer-9qhdm" (UID: "e3ef2522-f438-42f8-a961-b5b62e5c4836") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:52.925404 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.925019    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 18:45:52 +0000 UTC" deadline="2028-06-08 19:09:41.295846174 +0000 UTC"
Sep 15 18:50:52.925404 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:52.925303    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15168h18m48.370548683s"
Sep 15 18:50:53.093532 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.093432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:53.093532 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.093487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:53.093776 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:53.093588    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:50:53.094049 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:53.093985    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:50:53.107306 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.107222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9nr27" event={"ID":"2845410e-10e0-4248-862b-1d7bed1fd00d","Type":"ContainerStarted","Data":"55cab18ba12d7f7faf8bb54a9c1cf64fe1f1e1869137d373bf8960b7065b43c0"}
Sep 15 18:50:53.109202 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.109173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4gljt" event={"ID":"06644277-8f48-4af9-930e-c6493a57b420","Type":"ContainerStarted","Data":"9a82a60b005410ed0da25eeaf8c0220e6251ec666a061afe09d1a892efa4f236"}
Sep 15 18:50:53.122230 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.122203    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh" event={"ID":"96e24b14-fd77-449d-9ee6-6b7bb141a419","Type":"ContainerStarted","Data":"793ab2f0b515033ce10a6e4108410a62ecae47593b705cca14f269231cd9458c"}
Sep 15 18:50:53.129055 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.129022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8" event={"ID":"e94d481d-d5f0-481d-9b9b-b3924ae440fe","Type":"ContainerStarted","Data":"e3a114669b56623c0968d5503eaea416d7eab28419616bff2d67e84277a60f88"}
Sep 15 18:50:53.148527 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.148496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-srpw5" event={"ID":"b251128d-040c-49f9-8396-36a2db07a539","Type":"ContainerStarted","Data":"f826c710a1e2cbf2cc4bcd4cd430f50e430516e6e11a392e160fd0522b823071"}
Sep 15 18:50:53.160050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.160023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b75vj" event={"ID":"30cfe649-c99b-45ad-9325-c187e24073c3","Type":"ContainerStarted","Data":"24b72f8716a31ffd12cb640de4fc7b3638716228775d69ed79bb4a6c94878e17"}
Sep 15 18:50:53.181729 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.180342    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4glsv" event={"ID":"ac29fcda-9928-4ad9-9cee-6456cabf59a0","Type":"ContainerStarted","Data":"4f1c102997173ae353a6f0ae459fcb5031d3e08f6930e93b72382f7b3e459c16"}
Sep 15 18:50:53.185319 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.185290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7d7qs" event={"ID":"9e6b9574-0dfc-4290-b6ab-a785808d21a4","Type":"ContainerStarted","Data":"127973cf790f87fa2106b27f391fdbe8ccca66efffa857b94959200b1940d65b"}
Sep 15 18:50:53.187935 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.187910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6p5c5" event={"ID":"99b30109-dc43-48f1-a6d9-999575a5b20b","Type":"ContainerStarted","Data":"7a71df26516d5429660e98cc91c7e89f512a1d8320521ff0856f18af246cf7e0"}
Sep 15 18:50:53.506106 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.505472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbklb\" (UniqueName: \"kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb\") pod \"network-check-target-k8tcg\" (UID: \"b968fefe-3155-4c6c-8aee-c00f3af85ad0\") " pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:53.506106 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:53.505648    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:53.506106 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:53.505672    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:53.506106 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:53.505686    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nbklb for pod openshift-network-diagnostics/network-check-target-k8tcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:53.506106 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:53.505747    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb podName:b968fefe-3155-4c6c-8aee-c00f3af85ad0 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:55.505726661 +0000 UTC m=+6.047822176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbklb" (UniqueName: "kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb") pod "network-check-target-k8tcg" (UID: "b968fefe-3155-4c6c-8aee-c00f3af85ad0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:53.608201 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.608167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:53.608342 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.608242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:53.608420 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:53.608371    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:53.608478 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:53.608425    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs podName:eb8d5cee-7842-4f44-9079-94434a03ccff nodeName:}" failed. No retries permitted until 2026-09-15 18:50:55.608408472 +0000 UTC m=+6.150503981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs") pod "network-metrics-daemon-hktjq" (UID: "eb8d5cee-7842-4f44-9079-94434a03ccff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:53.608826 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:53.608806    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:53.608925 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:53.608867    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret podName:e3ef2522-f438-42f8-a961-b5b62e5c4836 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:55.608848847 +0000 UTC m=+6.150944359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret") pod "global-pull-secret-syncer-9qhdm" (UID: "e3ef2522-f438-42f8-a961-b5b62e5c4836") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:53.925535 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.925430    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 18:45:52 +0000 UTC" deadline="2028-03-09 07:37:45.862311494 +0000 UTC"
Sep 15 18:50:53.925535 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:53.925474    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12972h46m51.936842402s"
Sep 15 18:50:54.100330 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:54.099656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:54.100330 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:54.099787    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:50:54.221347 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:54.221136    2578 generic.go:356] "Generic (PLEG): container finished" podID="89c1bc29e834f6f6903ce15b6a1f531c" containerID="8e4217dee844f222feae9568a6a9c7fc63581ed2c9bb516077108f618d6c7a86" exitCode=0
Sep 15 18:50:54.221347 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:54.221191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal" event={"ID":"89c1bc29e834f6f6903ce15b6a1f531c","Type":"ContainerDied","Data":"8e4217dee844f222feae9568a6a9c7fc63581ed2c9bb516077108f618d6c7a86"}
Sep 15 18:50:55.094165 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:55.093272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:55.094165 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:55.093406    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:50:55.094165 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:55.094086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:55.109843 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:55.095337    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:50:55.242724 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:55.241977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal" event={"ID":"89c1bc29e834f6f6903ce15b6a1f531c","Type":"ContainerStarted","Data":"7ea14ecd63536869eb6edf2f8c188a5374424f2e02e3f8b5ea6e0acfae23730a"}
Sep 15 18:50:55.257751 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:55.257678    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-221.ec2.internal" podStartSLOduration=4.257655993 podStartE2EDuration="4.257655993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:55.256746884 +0000 UTC m=+5.798842417" watchObservedRunningTime="2026-09-15 18:50:55.257655993 +0000 UTC m=+5.799751526"
Sep 15 18:50:55.526320 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:55.526235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbklb\" (UniqueName: \"kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb\") pod \"network-check-target-k8tcg\" (UID: \"b968fefe-3155-4c6c-8aee-c00f3af85ad0\") " pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:55.526490 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:55.526439    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:55.526490 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:55.526463    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:55.526490 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:55.526476    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nbklb for pod openshift-network-diagnostics/network-check-target-k8tcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:55.526668 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:55.526533    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb podName:b968fefe-3155-4c6c-8aee-c00f3af85ad0 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:59.526514976 +0000 UTC m=+10.068610501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbklb" (UniqueName: "kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb") pod "network-check-target-k8tcg" (UID: "b968fefe-3155-4c6c-8aee-c00f3af85ad0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:55.627253 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:55.627215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:55.627458 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:55.627299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:55.627537 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:55.627456    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:55.627537 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:55.627519    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret podName:e3ef2522-f438-42f8-a961-b5b62e5c4836 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:59.627498605 +0000 UTC m=+10.169594117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret") pod "global-pull-secret-syncer-9qhdm" (UID: "e3ef2522-f438-42f8-a961-b5b62e5c4836") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:55.627960 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:55.627938    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:55.628083 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:55.627994    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs podName:eb8d5cee-7842-4f44-9079-94434a03ccff nodeName:}" failed. No retries permitted until 2026-09-15 18:50:59.627978327 +0000 UTC m=+10.170073838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs") pod "network-metrics-daemon-hktjq" (UID: "eb8d5cee-7842-4f44-9079-94434a03ccff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:56.095791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:56.095755    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:56.096293 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:56.095932    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:50:57.093931 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:57.093892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:57.094145 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:57.093909    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:57.094145 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:57.094045    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:50:57.094145 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:57.094105    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:50:58.093139 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:58.093106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:58.093645 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:58.093250    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:50:59.094063 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:59.093975    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:59.094506 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:59.094121    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:50:59.094694 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:59.094674    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:59.094866 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:59.094793    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:50:59.563421 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:59.562749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbklb\" (UniqueName: \"kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb\") pod \"network-check-target-k8tcg\" (UID: \"b968fefe-3155-4c6c-8aee-c00f3af85ad0\") " pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:50:59.563421 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:59.562947    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:59.563421 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:59.562969    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:59.563421 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:59.562982    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nbklb for pod openshift-network-diagnostics/network-check-target-k8tcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:59.563421 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:59.563039    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb podName:b968fefe-3155-4c6c-8aee-c00f3af85ad0 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:07.563019728 +0000 UTC m=+18.105115257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbklb" (UniqueName: "kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb") pod "network-check-target-k8tcg" (UID: "b968fefe-3155-4c6c-8aee-c00f3af85ad0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:59.663710 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:59.663432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:50:59.663710 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:50:59.663513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:50:59.663710 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:59.663649    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:59.663710 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:59.663713    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs podName:eb8d5cee-7842-4f44-9079-94434a03ccff nodeName:}" failed. No retries permitted until 2026-09-15 18:51:07.663694265 +0000 UTC m=+18.205789775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs") pod "network-metrics-daemon-hktjq" (UID: "eb8d5cee-7842-4f44-9079-94434a03ccff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:59.664066 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:59.663649    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:59.664123 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:50:59.664081    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret podName:e3ef2522-f438-42f8-a961-b5b62e5c4836 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:07.66406092 +0000 UTC m=+18.206156448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret") pod "global-pull-secret-syncer-9qhdm" (UID: "e3ef2522-f438-42f8-a961-b5b62e5c4836") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:00.094277 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:00.094244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:00.094878 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:00.094406    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:01.093229 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:01.092946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:01.093229 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:01.093118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:01.093229 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:01.093171    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:01.093229 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:01.093190    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:02.093328 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:02.093293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:02.093767 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:02.093423    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:03.092598 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:03.092563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:03.092759 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:03.092563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:03.092759 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:03.092666    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:03.092863 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:03.092764    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:04.093532 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:04.093495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:04.093987 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:04.093626    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:05.093443 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:05.093410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:05.093619 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:05.093542    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:05.093619 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:05.093602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:05.094051 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:05.093693    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:06.093205 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:06.093167    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:06.093372 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:06.093301    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:07.092681 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:07.092614    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:07.092681 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:07.092645    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:07.093214 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:07.092744    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:07.093214 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:07.092908    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:07.621593 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:07.621561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbklb\" (UniqueName: \"kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb\") pod \"network-check-target-k8tcg\" (UID: \"b968fefe-3155-4c6c-8aee-c00f3af85ad0\") " pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:07.621755 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:07.621692    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:07.621755 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:07.621706    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:07.621755 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:07.621716    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nbklb for pod openshift-network-diagnostics/network-check-target-k8tcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:07.621929 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:07.621760    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb podName:b968fefe-3155-4c6c-8aee-c00f3af85ad0 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:23.62174872 +0000 UTC m=+34.163844227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbklb" (UniqueName: "kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb") pod "network-check-target-k8tcg" (UID: "b968fefe-3155-4c6c-8aee-c00f3af85ad0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:07.722458 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:07.722432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:07.722569 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:07.722488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:07.722607 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:07.722581    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:07.722607 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:07.722577    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:07.722669 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:07.722627    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs podName:eb8d5cee-7842-4f44-9079-94434a03ccff nodeName:}" failed. No retries permitted until 2026-09-15 18:51:23.722614441 +0000 UTC m=+34.264709961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs") pod "network-metrics-daemon-hktjq" (UID: "eb8d5cee-7842-4f44-9079-94434a03ccff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:07.722669 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:07.722639    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret podName:e3ef2522-f438-42f8-a961-b5b62e5c4836 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:23.722633433 +0000 UTC m=+34.264728940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret") pod "global-pull-secret-syncer-9qhdm" (UID: "e3ef2522-f438-42f8-a961-b5b62e5c4836") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:08.093483 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:08.093449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:08.093903 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:08.093599    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:09.092748 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:09.092714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:09.092919 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:09.092755    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:09.092919 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:09.092840    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:09.093014 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:09.092972    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:10.093523 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:10.093491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:10.094001 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:10.093595    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:11.092985 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:11.092947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:11.092985 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:11.092983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:11.093221 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:11.093060    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:11.093221 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:11.093200    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:12.093201 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.093018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:12.093770 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:12.093247    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:12.276772 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.276726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal" event={"ID":"1f58418f2eb3a96678dd7e1f043ca38f","Type":"ContainerStarted","Data":"36d2fc37cc0322c2b6016d5f7b5063f60a5c47bdfc1a1d95c341de162b876447"}
Sep 15 18:51:12.278430 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.278393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9nr27" event={"ID":"2845410e-10e0-4248-862b-1d7bed1fd00d","Type":"ContainerStarted","Data":"e6e063e89eebc64cbd9841c5944d2412dbe7500e9a3eb155c8f4da9a472ce186"}
Sep 15 18:51:12.280176 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.280151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4gljt" event={"ID":"06644277-8f48-4af9-930e-c6493a57b420","Type":"ContainerStarted","Data":"b44cdcbb9d8f160be17a813d6a46c130d7ca2dea8e289059bd2bd889b06df10f"}
Sep 15 18:51:12.283766 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.283741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh" event={"ID":"96e24b14-fd77-449d-9ee6-6b7bb141a419","Type":"ContainerStarted","Data":"7660bbcfc6b96a82d61f64f883895a83362b7d9cff48600e7c0d4cf4ed16b1c9"}
Sep 15 18:51:12.283884 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.283828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh" event={"ID":"96e24b14-fd77-449d-9ee6-6b7bb141a419","Type":"ContainerStarted","Data":"c5e444526763e4aece345ff04324c4f28b219c309d48d3cea709acb41c61c71d"}
Sep 15 18:51:12.283884 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.283878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh" event={"ID":"96e24b14-fd77-449d-9ee6-6b7bb141a419","Type":"ContainerStarted","Data":"b6dce2ac7f23ad7cb91524e7238920dcc759856c5b00771918a5407c5db3c282"}
Sep 15 18:51:12.283968 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.283922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh" event={"ID":"96e24b14-fd77-449d-9ee6-6b7bb141a419","Type":"ContainerStarted","Data":"5c9ed010fbb64aaf75eccb5d6203a5a4ca0d563cd3e9771950d4b695cba441db"}
Sep 15 18:51:12.283968 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.283957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh" event={"ID":"96e24b14-fd77-449d-9ee6-6b7bb141a419","Type":"ContainerStarted","Data":"e4d15635e40371519880017e8ed7483a53e4e40316af8ddea6453b8d984bde84"}
Sep 15 18:51:12.284061 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.283998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh" event={"ID":"96e24b14-fd77-449d-9ee6-6b7bb141a419","Type":"ContainerStarted","Data":"634de9190f7d467935f3df782ee0d8dd68540153c7957ba2cef4ac2aacf1a28c"}
Sep 15 18:51:12.285324 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.285305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8" event={"ID":"e94d481d-d5f0-481d-9b9b-b3924ae440fe","Type":"ContainerStarted","Data":"960a10f86ae39b016026792305d7c3ef97fefb920dce1c143f79e9863babe09c"}
Sep 15 18:51:12.286908 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.286875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-srpw5" event={"ID":"b251128d-040c-49f9-8396-36a2db07a539","Type":"ContainerStarted","Data":"38a45089089d64cb3b5dd7016d843cbef8c1eb8f3e886cdfa2f2290b019b0b08"}
Sep 15 18:51:12.288477 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.288453    2578 generic.go:356] "Generic (PLEG): container finished" podID="30cfe649-c99b-45ad-9325-c187e24073c3" containerID="d5e905e1bf17470ec359dca76ba452e551f10fd0032c56e249187d4a9b0a2622" exitCode=0
Sep 15 18:51:12.288603 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.288517    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b75vj" event={"ID":"30cfe649-c99b-45ad-9325-c187e24073c3","Type":"ContainerDied","Data":"d5e905e1bf17470ec359dca76ba452e551f10fd0032c56e249187d4a9b0a2622"}
Sep 15 18:51:12.289392 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.289348    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-221.ec2.internal" podStartSLOduration=21.289332739 podStartE2EDuration="21.289332739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:12.288897092 +0000 UTC m=+22.830992615" watchObservedRunningTime="2026-09-15 18:51:12.289332739 +0000 UTC m=+22.831428269"
Sep 15 18:51:12.290431 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.290407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4glsv" event={"ID":"ac29fcda-9928-4ad9-9cee-6456cabf59a0","Type":"ContainerStarted","Data":"1a1d7208823b224511f4b96c6da71b20188ab9a94f6688d4b49cdeafb18b4f22"}
Sep 15 18:51:12.292651 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.292631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7d7qs" event={"ID":"9e6b9574-0dfc-4290-b6ab-a785808d21a4","Type":"ContainerStarted","Data":"db88aba92b509170866b6255ebf6e1426ff4bf7ef35bad3252e3f9e867a956a4"}
Sep 15 18:51:12.305814 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.305751    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4gljt" podStartSLOduration=2.975446226 podStartE2EDuration="22.305734382s" totalImagesPullingTime="19.330288156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:12.305091362 +0000 UTC m=+22.847186891" watchObservedRunningTime="2026-09-15 18:51:12.305734382 +0000 UTC m=+22.847829919"
Sep 15 18:51:12.353327 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.353244    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-srpw5" podStartSLOduration=3.045999803 podStartE2EDuration="22.353229707s" totalImagesPullingTime="19.307229904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:12.335881694 +0000 UTC m=+22.877977224" watchObservedRunningTime="2026-09-15 18:51:12.353229707 +0000 UTC m=+22.895325229"
Sep 15 18:51:12.369899 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.369857    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9nr27" podStartSLOduration=2.962228709 podStartE2EDuration="22.369840988s" totalImagesPullingTime="19.407612279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:12.355640194 +0000 UTC m=+22.897735721" watchObservedRunningTime="2026-09-15 18:51:12.369840988 +0000 UTC m=+22.911936519"
Sep 15 18:51:12.370125 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.370095    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4glsv" podStartSLOduration=3.034006431 podStartE2EDuration="22.370087159s" totalImagesPullingTime="19.336080728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:12.36974348 +0000 UTC m=+22.911839008" watchObservedRunningTime="2026-09-15 18:51:12.370087159 +0000 UTC m=+22.912182690"
Sep 15 18:51:12.930162 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.930132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:51:12.930694 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.930678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:51:12.943711 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:12.943672    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7d7qs" podStartSLOduration=3.584209802 podStartE2EDuration="22.943660049s" totalImagesPullingTime="19.359450247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:12.385775188 +0000 UTC m=+22.927870717" watchObservedRunningTime="2026-09-15 18:51:12.943660049 +0000 UTC m=+23.485755604"
Sep 15 18:51:13.092958 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.092927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:13.093081 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.092946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:13.093142 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:13.093071    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:13.093142 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:13.093089    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:13.180655 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.180628    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 18:51:13.297788 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.297691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8" event={"ID":"e94d481d-d5f0-481d-9b9b-b3924ae440fe","Type":"ContainerStarted","Data":"37f282399ca5bcfbaa63f285a0bca9db069a01f8e2088cce910d12430eb78ff7"}
Sep 15 18:51:13.299631 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.299555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6p5c5" event={"ID":"99b30109-dc43-48f1-a6d9-999575a5b20b","Type":"ContainerStarted","Data":"e04e8c0c39336c9d413a9af9078f54e13ace0836278db23585094bed9f0019d2"}
Sep 15 18:51:13.300614 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.300592    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:51:13.300888 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.300872    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-srpw5"
Sep 15 18:51:13.312875 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.312830    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6p5c5" podStartSLOduration=4.019766321 podStartE2EDuration="23.312816822s" totalImagesPullingTime="19.293050501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:13.312097053 +0000 UTC m=+23.854192582" watchObservedRunningTime="2026-09-15 18:51:13.312816822 +0000 UTC m=+23.854912361"
Sep 15 18:51:13.964924 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.964821    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T18:51:13.180645544Z","UUID":"63458f33-78e1-4fea-a573-20256154ee3d","Handler":null,"Name":"","Endpoint":""}
Sep 15 18:51:13.966749 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.966634    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 18:51:13.966749 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:13.966670    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 18:51:14.092737 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:14.092670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:14.092902 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:14.092819    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:14.307143 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:14.307107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh" event={"ID":"96e24b14-fd77-449d-9ee6-6b7bb141a419","Type":"ContainerStarted","Data":"9907c9b69c1b9ce35998e48b21e8d7eb9e23ecdf0325f33be3aeff9859f55239"}
Sep 15 18:51:15.093541 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:15.093512    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:15.093732 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:15.093640    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:15.093732 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:15.093699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:15.093854 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:15.093777    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:15.311835 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:15.311783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8" event={"ID":"e94d481d-d5f0-481d-9b9b-b3924ae440fe","Type":"ContainerStarted","Data":"59f1e32ee9c74ecf793096ff36c93c2c1f7cab9bb7da638744524445ba95671b"}
Sep 15 18:51:15.329031 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:15.328985    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnrq8" podStartSLOduration=3.597539789 podStartE2EDuration="25.328972279s" totalImagesPullingTime="21.73143249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:15.328704731 +0000 UTC m=+25.870800261" watchObservedRunningTime="2026-09-15 18:51:15.328972279 +0000 UTC m=+25.871067864"
Sep 15 18:51:16.093311 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:16.093282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:16.093489 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:16.093385    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:17.093668 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:17.093501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:17.094149 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:17.093501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:17.094149 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:17.093740    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:17.094149 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:17.093832    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:17.318760 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:17.318727    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh" event={"ID":"96e24b14-fd77-449d-9ee6-6b7bb141a419","Type":"ContainerStarted","Data":"825ceac0b4e5e14660b345299623d8d76e6ad26641f513dc414b8a909ffa6fbf"}
Sep 15 18:51:17.319017 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:17.318993    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:51:17.320676 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:17.320653    2578 generic.go:356] "Generic (PLEG): container finished" podID="30cfe649-c99b-45ad-9325-c187e24073c3" containerID="e1b75bfda39b4d24e16433aee4db8398bd95ccc0b0fe0e1e9c4574cc97acb213" exitCode=0
Sep 15 18:51:17.320840 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:17.320686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b75vj" event={"ID":"30cfe649-c99b-45ad-9325-c187e24073c3","Type":"ContainerDied","Data":"e1b75bfda39b4d24e16433aee4db8398bd95ccc0b0fe0e1e9c4574cc97acb213"}
Sep 15 18:51:17.334208 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:17.334160    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:51:17.344091 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:17.344053    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh" podStartSLOduration=7.944056304 podStartE2EDuration="27.34404238s" totalImagesPullingTime="19.399986076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:17.343668462 +0000 UTC m=+27.885763983" watchObservedRunningTime="2026-09-15 18:51:17.34404238 +0000 UTC m=+27.886137909"
Sep 15 18:51:18.093434 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:18.093402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:18.093690 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:18.093534    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:18.325175 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:18.325147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:51:18.325361 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:18.325343    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:51:18.341410 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:18.341360    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:51:18.514926 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:18.514894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9qhdm"]
Sep 15 18:51:18.515082 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:18.515017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:18.515133 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:18.515098    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:18.518217 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:18.518190    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k8tcg"]
Sep 15 18:51:18.518358 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:18.518288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:18.518424 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:18.518366    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:18.518962 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:18.518939    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hktjq"]
Sep 15 18:51:18.519051 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:18.519036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:18.519154 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:18.519132    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:19.328598 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:19.328426    2578 generic.go:356] "Generic (PLEG): container finished" podID="30cfe649-c99b-45ad-9325-c187e24073c3" containerID="69e4e50450e207f9dc96021252f3a607decc995a0c812477687d54c0c95ad1b4" exitCode=0
Sep 15 18:51:19.329182 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:19.328506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b75vj" event={"ID":"30cfe649-c99b-45ad-9325-c187e24073c3","Type":"ContainerDied","Data":"69e4e50450e207f9dc96021252f3a607decc995a0c812477687d54c0c95ad1b4"}
Sep 15 18:51:20.094347 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:20.094311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:20.094523 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:20.094398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:20.094523 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:20.094441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:20.094523 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:20.094444    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:20.094657 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:20.094520    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:20.094657 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:20.094619    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:21.335524 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:21.335495    2578 generic.go:356] "Generic (PLEG): container finished" podID="30cfe649-c99b-45ad-9325-c187e24073c3" containerID="f513ec0567a556476de136b51242a09ffeb48a1307be30e02db77d8feb53a010" exitCode=0
Sep 15 18:51:21.335891 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:21.335540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b75vj" event={"ID":"30cfe649-c99b-45ad-9325-c187e24073c3","Type":"ContainerDied","Data":"f513ec0567a556476de136b51242a09ffeb48a1307be30e02db77d8feb53a010"}
Sep 15 18:51:22.092877 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:22.092842    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:22.093054 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:22.092842    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:22.093054 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:22.092967    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:22.093054 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:22.092997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:22.093205 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:22.093115    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:22.093260 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:22.093217    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:23.630673 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:23.630630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbklb\" (UniqueName: \"kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb\") pod \"network-check-target-k8tcg\" (UID: \"b968fefe-3155-4c6c-8aee-c00f3af85ad0\") " pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:23.631133 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:23.630871    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:23.631133 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:23.630891    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:23.631133 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:23.630901    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nbklb for pod openshift-network-diagnostics/network-check-target-k8tcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:23.631133 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:23.630996    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb podName:b968fefe-3155-4c6c-8aee-c00f3af85ad0 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:55.630952857 +0000 UTC m=+66.173048364 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbklb" (UniqueName: "kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb") pod "network-check-target-k8tcg" (UID: "b968fefe-3155-4c6c-8aee-c00f3af85ad0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:23.731296 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:23.731265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:23.731451 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:23.731328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:23.731451 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:23.731403    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:23.731451 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:23.731441    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:23.731590 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:23.731460    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret podName:e3ef2522-f438-42f8-a961-b5b62e5c4836 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:55.7314457 +0000 UTC m=+66.273541206 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret") pod "global-pull-secret-syncer-9qhdm" (UID: "e3ef2522-f438-42f8-a961-b5b62e5c4836") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:23.731590 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:23.731488    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs podName:eb8d5cee-7842-4f44-9079-94434a03ccff nodeName:}" failed. No retries permitted until 2026-09-15 18:51:55.731472754 +0000 UTC m=+66.273568274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs") pod "network-metrics-daemon-hktjq" (UID: "eb8d5cee-7842-4f44-9079-94434a03ccff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:24.093596 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.093519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:24.093757 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.093536    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:24.093757 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:24.093637    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k8tcg" podUID="b968fefe-3155-4c6c-8aee-c00f3af85ad0"
Sep 15 18:51:24.093757 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.093649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:24.093757 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:24.093743    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9qhdm" podUID="e3ef2522-f438-42f8-a961-b5b62e5c4836"
Sep 15 18:51:24.093999 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:24.093865    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:51:24.313512 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.313483    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-221.ec2.internal" event="NodeReady"
Sep 15 18:51:24.313678 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.313617    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 18:51:24.377230 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.377197    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bngct"]
Sep 15 18:51:24.417835 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.417788    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bhb6b"]
Sep 15 18:51:24.417969 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.417921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.420376 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.420354    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 18:51:24.420490 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.420391    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hj774\""
Sep 15 18:51:24.420606 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.420597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 18:51:24.437719 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.437695    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bhb6b"]
Sep 15 18:51:24.437834 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.437738    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bngct"]
Sep 15 18:51:24.437834 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.437816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bhb6b"
Sep 15 18:51:24.440391 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.440365    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 18:51:24.440504 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.440425    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xmcwz\""
Sep 15 18:51:24.440620 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.440575    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 18:51:24.440691 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.440622    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 18:51:24.538210 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.538179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.538210 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.538216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd544b8-ee8e-4117-9103-d67234994f82-cert\") pod \"ingress-canary-bhb6b\" (UID: \"9cd544b8-ee8e-4117-9103-d67234994f82\") " pod="openshift-ingress-canary/ingress-canary-bhb6b"
Sep 15 18:51:24.538410 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.538291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d53002-a924-4ea1-a656-da4e6ba1e6db-tmp-dir\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.538410 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.538327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwdd\" (UniqueName: \"kubernetes.io/projected/05d53002-a924-4ea1-a656-da4e6ba1e6db-kube-api-access-vqwdd\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.538410 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.538351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkf9\" (UniqueName: \"kubernetes.io/projected/9cd544b8-ee8e-4117-9103-d67234994f82-kube-api-access-gzkf9\") pod \"ingress-canary-bhb6b\" (UID: \"9cd544b8-ee8e-4117-9103-d67234994f82\") " pod="openshift-ingress-canary/ingress-canary-bhb6b"
Sep 15 18:51:24.538519 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.538415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05d53002-a924-4ea1-a656-da4e6ba1e6db-config-volume\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.639598 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.639509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d53002-a924-4ea1-a656-da4e6ba1e6db-tmp-dir\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.639598 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.639561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwdd\" (UniqueName: \"kubernetes.io/projected/05d53002-a924-4ea1-a656-da4e6ba1e6db-kube-api-access-vqwdd\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.639598 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.639579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkf9\" (UniqueName: \"kubernetes.io/projected/9cd544b8-ee8e-4117-9103-d67234994f82-kube-api-access-gzkf9\") pod \"ingress-canary-bhb6b\" (UID: \"9cd544b8-ee8e-4117-9103-d67234994f82\") " pod="openshift-ingress-canary/ingress-canary-bhb6b"
Sep 15 18:51:24.640155 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.639621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05d53002-a924-4ea1-a656-da4e6ba1e6db-config-volume\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.640155 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.639657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.640155 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.639681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd544b8-ee8e-4117-9103-d67234994f82-cert\") pod \"ingress-canary-bhb6b\" (UID: \"9cd544b8-ee8e-4117-9103-d67234994f82\") " pod="openshift-ingress-canary/ingress-canary-bhb6b"
Sep 15 18:51:24.640155 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:24.639828    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:24.640155 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:24.639898    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls podName:05d53002-a924-4ea1-a656-da4e6ba1e6db nodeName:}" failed. No retries permitted until 2026-09-15 18:51:25.139876874 +0000 UTC m=+35.681972387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls") pod "dns-default-bngct" (UID: "05d53002-a924-4ea1-a656-da4e6ba1e6db") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:24.640155 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.639967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d53002-a924-4ea1-a656-da4e6ba1e6db-tmp-dir\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.640402 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.640207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05d53002-a924-4ea1-a656-da4e6ba1e6db-config-volume\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.644363 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.644336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd544b8-ee8e-4117-9103-d67234994f82-cert\") pod \"ingress-canary-bhb6b\" (UID: \"9cd544b8-ee8e-4117-9103-d67234994f82\") " pod="openshift-ingress-canary/ingress-canary-bhb6b"
Sep 15 18:51:24.651886 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.651836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwdd\" (UniqueName: \"kubernetes.io/projected/05d53002-a924-4ea1-a656-da4e6ba1e6db-kube-api-access-vqwdd\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:24.652249 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.652228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkf9\" (UniqueName: \"kubernetes.io/projected/9cd544b8-ee8e-4117-9103-d67234994f82-kube-api-access-gzkf9\") pod \"ingress-canary-bhb6b\" (UID: \"9cd544b8-ee8e-4117-9103-d67234994f82\") " pod="openshift-ingress-canary/ingress-canary-bhb6b"
Sep 15 18:51:24.747116 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:24.747083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bhb6b"
Sep 15 18:51:25.143625 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:25.143596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:25.143815 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:25.143724    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:25.143815 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:25.143812    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls podName:05d53002-a924-4ea1-a656-da4e6ba1e6db nodeName:}" failed. No retries permitted until 2026-09-15 18:51:26.143775337 +0000 UTC m=+36.685870858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls") pod "dns-default-bngct" (UID: "05d53002-a924-4ea1-a656-da4e6ba1e6db") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:25.566489 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:25.566292    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bhb6b"]
Sep 15 18:51:25.570707 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:51:25.570681    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd544b8_ee8e_4117_9103_d67234994f82.slice/crio-992dacbdbc105481cdd2235d2d38c3701565e31fd85e0f4bc06b5d03de614c2c WatchSource:0}: Error finding container 992dacbdbc105481cdd2235d2d38c3701565e31fd85e0f4bc06b5d03de614c2c: Status 404 returned error can't find the container with id 992dacbdbc105481cdd2235d2d38c3701565e31fd85e0f4bc06b5d03de614c2c
Sep 15 18:51:26.093385 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.093304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:26.093951 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.093304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:26.093951 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.093313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:26.095829 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.095809    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 18:51:26.096634 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.096606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 18:51:26.096715 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.096632    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mrmrg\""
Sep 15 18:51:26.096715 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.096655    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bc8c\""
Sep 15 18:51:26.096715 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.096682    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 18:51:26.096715 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.096692    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 18:51:26.148504 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.148480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:26.148594 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:26.148576    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:26.148633 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:26.148629    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls podName:05d53002-a924-4ea1-a656-da4e6ba1e6db nodeName:}" failed. No retries permitted until 2026-09-15 18:51:28.148615862 +0000 UTC m=+38.690711369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls") pod "dns-default-bngct" (UID: "05d53002-a924-4ea1-a656-da4e6ba1e6db") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:26.350226 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.350137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bhb6b" event={"ID":"9cd544b8-ee8e-4117-9103-d67234994f82","Type":"ContainerStarted","Data":"992dacbdbc105481cdd2235d2d38c3701565e31fd85e0f4bc06b5d03de614c2c"}
Sep 15 18:51:26.353420 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.353387    2578 generic.go:356] "Generic (PLEG): container finished" podID="30cfe649-c99b-45ad-9325-c187e24073c3" containerID="d533305c3574b1841a8e6f8a08e42c68c801176814abc8a79edc8d71056e5ba5" exitCode=0
Sep 15 18:51:26.353545 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:26.353431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b75vj" event={"ID":"30cfe649-c99b-45ad-9325-c187e24073c3","Type":"ContainerDied","Data":"d533305c3574b1841a8e6f8a08e42c68c801176814abc8a79edc8d71056e5ba5"}
Sep 15 18:51:27.360994 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:27.360953    2578 generic.go:356] "Generic (PLEG): container finished" podID="30cfe649-c99b-45ad-9325-c187e24073c3" containerID="b41b99b7d5393f2fd0c587d2b06c6cd8fe3129b26bd3a50080ab37a58ee27c19" exitCode=0
Sep 15 18:51:27.361563 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:27.361030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b75vj" event={"ID":"30cfe649-c99b-45ad-9325-c187e24073c3","Type":"ContainerDied","Data":"b41b99b7d5393f2fd0c587d2b06c6cd8fe3129b26bd3a50080ab37a58ee27c19"}
Sep 15 18:51:28.163005 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:28.162773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:28.163143 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:28.162927    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:28.163143 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:28.163098    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls podName:05d53002-a924-4ea1-a656-da4e6ba1e6db nodeName:}" failed. No retries permitted until 2026-09-15 18:51:32.163085184 +0000 UTC m=+42.705180691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls") pod "dns-default-bngct" (UID: "05d53002-a924-4ea1-a656-da4e6ba1e6db") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:28.367053 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:28.366973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b75vj" event={"ID":"30cfe649-c99b-45ad-9325-c187e24073c3","Type":"ContainerStarted","Data":"f4919f6e77bc109d6b941c8ee9cbe938c489f7dfe53276f4acee7d277e691c98"}
Sep 15 18:51:28.368459 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:28.368439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bhb6b" event={"ID":"9cd544b8-ee8e-4117-9103-d67234994f82","Type":"ContainerStarted","Data":"7ea96aaff346f89eee9ef67ca69f4c0074b4fce1568569621d049e943da27dff"}
Sep 15 18:51:28.387677 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:28.387628    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b75vj" podStartSLOduration=8.119953858 podStartE2EDuration="38.387611059s" totalImagesPullingTime="30.267657201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:28.386299533 +0000 UTC m=+38.928395062" watchObservedRunningTime="2026-09-15 18:51:28.387611059 +0000 UTC m=+38.929706589"
Sep 15 18:51:28.401490 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:28.401449    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bhb6b" podStartSLOduration=2.149050164 podStartE2EDuration="4.401437292s" totalImagesPullingTime="2.252387128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:28.401098018 +0000 UTC m=+38.943193547" watchObservedRunningTime="2026-09-15 18:51:28.401437292 +0000 UTC m=+38.943532821"
Sep 15 18:51:32.189033 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:32.188989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:32.189415 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:32.189143    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:32.189415 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:32.189202    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls podName:05d53002-a924-4ea1-a656-da4e6ba1e6db nodeName:}" failed. No retries permitted until 2026-09-15 18:51:40.189187003 +0000 UTC m=+50.731282515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls") pod "dns-default-bngct" (UID: "05d53002-a924-4ea1-a656-da4e6ba1e6db") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:40.240342 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:40.240307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:40.240757 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:40.240423    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:40.240757 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:40.240474    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls podName:05d53002-a924-4ea1-a656-da4e6ba1e6db nodeName:}" failed. No retries permitted until 2026-09-15 18:51:56.240460561 +0000 UTC m=+66.782556067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls") pod "dns-default-bngct" (UID: "05d53002-a924-4ea1-a656-da4e6ba1e6db") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:50.367813 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:50.367770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qg5zh"
Sep 15 18:51:55.644017 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.643965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbklb\" (UniqueName: \"kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb\") pod \"network-check-target-k8tcg\" (UID: \"b968fefe-3155-4c6c-8aee-c00f3af85ad0\") " pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:55.646505 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.646487    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 18:51:55.656235 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.656217    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 18:51:55.668357 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.668326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbklb\" (UniqueName: \"kubernetes.io/projected/b968fefe-3155-4c6c-8aee-c00f3af85ad0-kube-api-access-nbklb\") pod \"network-check-target-k8tcg\" (UID: \"b968fefe-3155-4c6c-8aee-c00f3af85ad0\") " pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:55.744253 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.744230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:55.744374 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.744276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:51:55.746829 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.746810    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 18:51:55.746943 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.746855    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 18:51:55.754569 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:55.754553    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 18:51:55.754645 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:55.754606    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs podName:eb8d5cee-7842-4f44-9079-94434a03ccff nodeName:}" failed. No retries permitted until 2026-09-15 18:52:59.754591225 +0000 UTC m=+130.296686731 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs") pod "network-metrics-daemon-hktjq" (UID: "eb8d5cee-7842-4f44-9079-94434a03ccff") : secret "metrics-daemon-secret" not found
Sep 15 18:51:55.756479 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.756453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3ef2522-f438-42f8-a961-b5b62e5c4836-original-pull-secret\") pod \"global-pull-secret-syncer-9qhdm\" (UID: \"e3ef2522-f438-42f8-a961-b5b62e5c4836\") " pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:55.804672 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.804646    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mrmrg\""
Sep 15 18:51:55.808372 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.808356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9qhdm"
Sep 15 18:51:55.813358 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:55.813343    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:51:56.080240 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:56.080158    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9qhdm"]
Sep 15 18:51:56.083401 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:51:56.083370    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ef2522_f438_42f8_a961_b5b62e5c4836.slice/crio-8811c91025edea57b81cebd286eed6dc834a18e15d850a648f2301a403fd2300 WatchSource:0}: Error finding container 8811c91025edea57b81cebd286eed6dc834a18e15d850a648f2301a403fd2300: Status 404 returned error can't find the container with id 8811c91025edea57b81cebd286eed6dc834a18e15d850a648f2301a403fd2300
Sep 15 18:51:56.083505 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:56.083428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k8tcg"]
Sep 15 18:51:56.086561 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:51:56.086542    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb968fefe_3155_4c6c_8aee_c00f3af85ad0.slice/crio-4795f7d3463396da4900f0cb2903ecc2c4bf82eed82f442d25248ba9ee7c367d WatchSource:0}: Error finding container 4795f7d3463396da4900f0cb2903ecc2c4bf82eed82f442d25248ba9ee7c367d: Status 404 returned error can't find the container with id 4795f7d3463396da4900f0cb2903ecc2c4bf82eed82f442d25248ba9ee7c367d
Sep 15 18:51:56.248064 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:56.248032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:51:56.248212 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:56.248145    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:56.248212 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:51:56.248194    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls podName:05d53002-a924-4ea1-a656-da4e6ba1e6db nodeName:}" failed. No retries permitted until 2026-09-15 18:52:28.248179545 +0000 UTC m=+98.790275051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls") pod "dns-default-bngct" (UID: "05d53002-a924-4ea1-a656-da4e6ba1e6db") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:56.434168 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:56.434137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9qhdm" event={"ID":"e3ef2522-f438-42f8-a961-b5b62e5c4836","Type":"ContainerStarted","Data":"8811c91025edea57b81cebd286eed6dc834a18e15d850a648f2301a403fd2300"}
Sep 15 18:51:56.435207 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:51:56.435185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k8tcg" event={"ID":"b968fefe-3155-4c6c-8aee-c00f3af85ad0","Type":"ContainerStarted","Data":"4795f7d3463396da4900f0cb2903ecc2c4bf82eed82f442d25248ba9ee7c367d"}
Sep 15 18:52:01.449912 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:01.449872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9qhdm" event={"ID":"e3ef2522-f438-42f8-a961-b5b62e5c4836","Type":"ContainerStarted","Data":"0beac4a8f585faf29ea3f2f9e08b722390ebee7b5ec253f46319856411f9e94d"}
Sep 15 18:52:01.451380 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:01.451356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k8tcg" event={"ID":"b968fefe-3155-4c6c-8aee-c00f3af85ad0","Type":"ContainerStarted","Data":"319298a5a94c65c549dff0749cbe192ed2fac7ca51854f0c58cd668a5efc87f2"}
Sep 15 18:52:01.451496 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:01.451484    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:52:01.464323 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:01.464283    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9qhdm" podStartSLOduration=66.587857999 podStartE2EDuration="1m11.464271226s" totalImagesPullingTime="4.876413227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:01.463445243 +0000 UTC m=+72.005540774" watchObservedRunningTime="2026-09-15 18:52:01.464271226 +0000 UTC m=+72.006366755"
Sep 15 18:52:01.477012 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:01.476974    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k8tcg" podStartSLOduration=66.603843286 podStartE2EDuration="1m11.476959713s" totalImagesPullingTime="4.873116427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:01.476472852 +0000 UTC m=+72.018568372" watchObservedRunningTime="2026-09-15 18:52:01.476959713 +0000 UTC m=+72.019055242"
Sep 15 18:52:28.251830 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:28.251772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:52:28.252233 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:52:28.251915    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:52:28.252233 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:52:28.251984    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls podName:05d53002-a924-4ea1-a656-da4e6ba1e6db nodeName:}" failed. No retries permitted until 2026-09-15 18:53:32.251969536 +0000 UTC m=+162.794065043 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls") pod "dns-default-bngct" (UID: "05d53002-a924-4ea1-a656-da4e6ba1e6db") : secret "dns-default-metrics-tls" not found
Sep 15 18:52:32.455613 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:32.455582    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k8tcg"
Sep 15 18:52:34.200534 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.200501    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"]
Sep 15 18:52:34.204005 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.203945    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"]
Sep 15 18:52:34.204122 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.204067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"
Sep 15 18:52:34.206060 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.206042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.206502 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.206483    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 18:52:34.206502 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.206499    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8l5l2\""
Sep 15 18:52:34.207249 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.207235    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 18:52:34.207310 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.207235    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 15 18:52:34.207310 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.207298    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 18:52:34.208017 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.207997    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 15 18:52:34.208118 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.208025    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 15 18:52:34.208185 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.208144    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 15 18:52:34.208422 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.208407    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 15 18:52:34.213474 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.213456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"]
Sep 15 18:52:34.219503 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.219478    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"]
Sep 15 18:52:34.289940 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.289919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdfqd\" (UniqueName: \"kubernetes.io/projected/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-kube-api-access-bdfqd\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.290057 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.289948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1db21d82-5d76-47e8-8d2e-e7957fdfda9e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk\" (UID: \"1db21d82-5d76-47e8-8d2e-e7957fdfda9e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"
Sep 15 18:52:34.290057 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.289968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.290057 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.289987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnvk\" (UniqueName: \"kubernetes.io/projected/1db21d82-5d76-47e8-8d2e-e7957fdfda9e-kube-api-access-clnvk\") pod \"managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk\" (UID: \"1db21d82-5d76-47e8-8d2e-e7957fdfda9e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"
Sep 15 18:52:34.290161 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.290062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-ca\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.290161 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.290134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.290161 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.290155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-hub\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.290254 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.290169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.390545 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.390519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.390607 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.390552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-hub\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.390607 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.390568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.390607 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.390595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdfqd\" (UniqueName: \"kubernetes.io/projected/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-kube-api-access-bdfqd\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.390750 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.390610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1db21d82-5d76-47e8-8d2e-e7957fdfda9e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk\" (UID: \"1db21d82-5d76-47e8-8d2e-e7957fdfda9e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"
Sep 15 18:52:34.390750 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.390627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.390750 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.390653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnvk\" (UniqueName: \"kubernetes.io/projected/1db21d82-5d76-47e8-8d2e-e7957fdfda9e-kube-api-access-clnvk\") pod \"managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk\" (UID: \"1db21d82-5d76-47e8-8d2e-e7957fdfda9e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"
Sep 15 18:52:34.390750 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.390685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-ca\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.391442 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.391420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.393263 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.393235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.393343 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.393319    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-ca\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.393387 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.393347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-hub\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.393593 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.393576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.393726 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.393705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1db21d82-5d76-47e8-8d2e-e7957fdfda9e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk\" (UID: \"1db21d82-5d76-47e8-8d2e-e7957fdfda9e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"
Sep 15 18:52:34.397769 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.397745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnvk\" (UniqueName: \"kubernetes.io/projected/1db21d82-5d76-47e8-8d2e-e7957fdfda9e-kube-api-access-clnvk\") pod \"managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk\" (UID: \"1db21d82-5d76-47e8-8d2e-e7957fdfda9e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"
Sep 15 18:52:34.397991 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.397974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdfqd\" (UniqueName: \"kubernetes.io/projected/5ce5c596-e80c-4907-bef1-9d0c360bdfcd-kube-api-access-bdfqd\") pod \"cluster-proxy-proxy-agent-77cf6bc965-r5w5h\" (UID: \"5ce5c596-e80c-4907-bef1-9d0c360bdfcd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.519258 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.519192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"
Sep 15 18:52:34.525418 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.525385    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:52:34.787215 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.787149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk"]
Sep 15 18:52:34.790733 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:52:34.790704    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1db21d82_5d76_47e8_8d2e_e7957fdfda9e.slice/crio-383a204ede9feab7f140ff8cd1de9c4e97eea1ddfcc9f143033a67cd60e48d2e WatchSource:0}: Error finding container 383a204ede9feab7f140ff8cd1de9c4e97eea1ddfcc9f143033a67cd60e48d2e: Status 404 returned error can't find the container with id 383a204ede9feab7f140ff8cd1de9c4e97eea1ddfcc9f143033a67cd60e48d2e
Sep 15 18:52:34.795265 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:34.795180    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"]
Sep 15 18:52:34.799128 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:52:34.799106    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ce5c596_e80c_4907_bef1_9d0c360bdfcd.slice/crio-845ffdc565779d258bd1f4b4460f9ec1c80108cad428e13c27aed2c40e183787 WatchSource:0}: Error finding container 845ffdc565779d258bd1f4b4460f9ec1c80108cad428e13c27aed2c40e183787: Status 404 returned error can't find the container with id 845ffdc565779d258bd1f4b4460f9ec1c80108cad428e13c27aed2c40e183787
Sep 15 18:52:35.531247 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:35.531208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" event={"ID":"5ce5c596-e80c-4907-bef1-9d0c360bdfcd","Type":"ContainerStarted","Data":"845ffdc565779d258bd1f4b4460f9ec1c80108cad428e13c27aed2c40e183787"}
Sep 15 18:52:35.532335 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:35.532312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk" event={"ID":"1db21d82-5d76-47e8-8d2e-e7957fdfda9e","Type":"ContainerStarted","Data":"383a204ede9feab7f140ff8cd1de9c4e97eea1ddfcc9f143033a67cd60e48d2e"}
Sep 15 18:52:40.545968 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:40.545879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk" event={"ID":"1db21d82-5d76-47e8-8d2e-e7957fdfda9e","Type":"ContainerStarted","Data":"e5090857b043cc6e2f450b632f484f254037ab1a0df24efcff534c3b26d734c1"}
Sep 15 18:52:40.560623 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:40.560578    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk" podStartSLOduration=1.15000341 podStartE2EDuration="6.560563632s" totalImagesPullingTime="5.410560222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:40.559416019 +0000 UTC m=+111.101511546" watchObservedRunningTime="2026-09-15 18:52:40.560563632 +0000 UTC m=+111.102659161"
Sep 15 18:52:49.570260 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:49.570225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" event={"ID":"5ce5c596-e80c-4907-bef1-9d0c360bdfcd","Type":"ContainerStarted","Data":"75108dee8d6c932ad4ff4a545ad22c536b3ca8d89636be543e498273c981c763"}
Sep 15 18:52:49.570627 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:49.570269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" event={"ID":"5ce5c596-e80c-4907-bef1-9d0c360bdfcd","Type":"ContainerStarted","Data":"3d219c3e8aaba414f37b09b6e3a2791f859d18fc0d25d0e5aa0459e0136aa73e"}
Sep 15 18:52:49.570627 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:49.570282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" event={"ID":"5ce5c596-e80c-4907-bef1-9d0c360bdfcd","Type":"ContainerStarted","Data":"45814f703eed4a67ffdb9d774068232bc2e47cdf72e3690ee99cd5cd7d502c3e"}
Sep 15 18:52:49.587596 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:49.587547    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" podStartSLOduration=1.687886458 podStartE2EDuration="15.587529549s" totalImagesPullingTime="13.899643091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:49.586698475 +0000 UTC m=+120.128794007" watchObservedRunningTime="2026-09-15 18:52:49.587529549 +0000 UTC m=+120.129625079"
Sep 15 18:52:59.758061 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:52:59.758009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:52:59.758487 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:52:59.758126    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 18:52:59.758487 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:52:59.758193    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs podName:eb8d5cee-7842-4f44-9079-94434a03ccff nodeName:}" failed. No retries permitted until 2026-09-15 18:55:01.75817969 +0000 UTC m=+252.300275202 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs") pod "network-metrics-daemon-hktjq" (UID: "eb8d5cee-7842-4f44-9079-94434a03ccff") : secret "metrics-daemon-secret" not found
Sep 15 18:53:27.431127 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:53:27.431084    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bngct" podUID="05d53002-a924-4ea1-a656-da4e6ba1e6db"
Sep 15 18:53:27.679026 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:53:27.678997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bngct"
Sep 15 18:53:29.113418 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:53:29.113363    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:53:32.275701 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:53:32.275653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:53:32.276154 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:53:32.275792    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:53:32.276154 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:53:32.275871    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls podName:05d53002-a924-4ea1-a656-da4e6ba1e6db nodeName:}" failed. No retries permitted until 2026-09-15 18:55:34.275856762 +0000 UTC m=+284.817952269 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls") pod "dns-default-bngct" (UID: "05d53002-a924-4ea1-a656-da4e6ba1e6db") : secret "dns-default-metrics-tls" not found
Sep 15 18:53:40.096434 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:53:40.096399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:54:40.885892 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:40.885859    2578 generic.go:356] "Generic (PLEG): container finished" podID="1db21d82-5d76-47e8-8d2e-e7957fdfda9e" containerID="e5090857b043cc6e2f450b632f484f254037ab1a0df24efcff534c3b26d734c1" exitCode=255
Sep 15 18:54:40.886406 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:40.885914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk" event={"ID":"1db21d82-5d76-47e8-8d2e-e7957fdfda9e","Type":"ContainerDied","Data":"e5090857b043cc6e2f450b632f484f254037ab1a0df24efcff534c3b26d734c1"}
Sep 15 18:54:40.886406 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:40.886262    2578 scope.go:117] "RemoveContainer" containerID="e5090857b043cc6e2f450b632f484f254037ab1a0df24efcff534c3b26d734c1"
Sep 15 18:54:41.890479 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:41.890444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f8d6d5dc6-62sxk" event={"ID":"1db21d82-5d76-47e8-8d2e-e7957fdfda9e","Type":"ContainerStarted","Data":"6ea84692502071634d0fab73da6e832db412a1167d78e84aec6d2501930bcb1e"}
Sep 15 18:54:58.184857 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.184825    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hn444"]
Sep 15 18:54:58.187610 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.187591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.190198 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.190166    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 18:54:58.190299 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.190234    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tvqm8\""
Sep 15 18:54:58.190547 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.190531    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 18:54:58.191066 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.191048    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 18:54:58.191119 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.191073    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 18:54:58.198290 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.198267    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hn444"]
Sep 15 18:54:58.240196 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.240167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ac0fda6-2405-4bcf-a74b-612af3c2268c-crio-socket\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.240305 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.240221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb2n\" (UniqueName: \"kubernetes.io/projected/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-api-access-8mb2n\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.240305 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.240273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.240305 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.240295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ac0fda6-2405-4bcf-a74b-612af3c2268c-data-volume\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.240421 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.240310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.341158 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.341130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ac0fda6-2405-4bcf-a74b-612af3c2268c-data-volume\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.341158 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.341157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.341329 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.341175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ac0fda6-2405-4bcf-a74b-612af3c2268c-crio-socket\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.341329 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.341216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb2n\" (UniqueName: \"kubernetes.io/projected/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-api-access-8mb2n\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.341329 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.341248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.341329 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:58.341256    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.341329 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.341323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ac0fda6-2405-4bcf-a74b-612af3c2268c-crio-socket\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.341498 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:58.341339    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:54:58.841316868 +0000 UTC m=+249.383412399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.341498 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:58.341348    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.341498 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:58.341396    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:54:58.84138411 +0000 UTC m=+249.383479620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.341498 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.341492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ac0fda6-2405-4bcf-a74b-612af3c2268c-data-volume\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.349952 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.349929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb2n\" (UniqueName: \"kubernetes.io/projected/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-api-access-8mb2n\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.845197 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.845161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.845373 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:58.845204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:58.845373 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:58.845289    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.845373 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:58.845334    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.845373 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:58.845359    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:54:59.845338788 +0000 UTC m=+250.387434318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.845515 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:58.845418    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:54:59.845405397 +0000 UTC m=+250.387500910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:59.851715 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:59.851675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:59.852122 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:54:59.851752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:54:59.852122 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:59.851788    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:54:59.852122 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:59.851851    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:59.852122 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:59.851873    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:55:01.851858643 +0000 UTC m=+252.393954151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : configmap "kube-rbac-proxy" not found
Sep 15 18:54:59.852122 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:54:59.851898    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:55:01.851887905 +0000 UTC m=+252.393983412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:00.695989 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:00.695962    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4gljt_06644277-8f48-4af9-930e-c6493a57b420/dns-node-resolver/0.log"
Sep 15 18:55:01.698477 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:01.698453    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4glsv_ac29fcda-9928-4ad9-9cee-6456cabf59a0/node-ca/0.log"
Sep 15 18:55:01.767874 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:01.767847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:55:01.767969 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:01.767960    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 18:55:01.768018 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:01.768009    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs podName:eb8d5cee-7842-4f44-9079-94434a03ccff nodeName:}" failed. No retries permitted until 2026-09-15 18:57:03.767995779 +0000 UTC m=+374.310091286 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs") pod "network-metrics-daemon-hktjq" (UID: "eb8d5cee-7842-4f44-9079-94434a03ccff") : secret "metrics-daemon-secret" not found
Sep 15 18:55:01.868626 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:01.868577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:55:01.868626 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:01.868635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:55:01.868777 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:01.868706    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:55:01.868777 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:01.868721    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:01.868777 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:01.868756    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:55:05.868742812 +0000 UTC m=+256.410838319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : configmap "kube-rbac-proxy" not found
Sep 15 18:55:01.868777 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:01.868774    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:55:05.868761509 +0000 UTC m=+256.410857016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:02.696489 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:02.696437    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bhb6b_9cd544b8-ee8e-4117-9103-d67234994f82/serve-healthcheck-canary/0.log"
Sep 15 18:55:05.898337 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:05.898291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:55:05.898960 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:05.898344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:55:05.898960 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:05.898436    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:55:05.898960 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:05.898453    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:05.898960 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:05.898508    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:55:13.898486955 +0000 UTC m=+264.440582465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : configmap "kube-rbac-proxy" not found
Sep 15 18:55:05.898960 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:05.898526    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:55:13.898517849 +0000 UTC m=+264.440613358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:13.956289 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:13.956198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:55:13.956289 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:13.956277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:55:13.956860 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:13.956364    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:13.956860 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:13.956426    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls podName:3ac0fda6-2405-4bcf-a74b-612af3c2268c nodeName:}" failed. No retries permitted until 2026-09-15 18:55:29.956413182 +0000 UTC m=+280.498508694 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hn444" (UID: "3ac0fda6-2405-4bcf-a74b-612af3c2268c") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:13.956860 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:13.956701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ac0fda6-2405-4bcf-a74b-612af3c2268c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:55:25.745126 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.745091    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5478c48966-jtvh2"]
Sep 15 18:55:25.747911 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.747888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.750099 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.750065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 18:55:25.750237 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.750150    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 18:55:25.750237 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.750223    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 18:55:25.750924 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.750903    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 18:55:25.751014 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.750952    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kbhdv\""
Sep 15 18:55:25.751184 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.751168    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 18:55:25.751258 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.751176    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 18:55:25.751258 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.751196    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 18:55:25.758135 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.758113    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5478c48966-jtvh2"]
Sep 15 18:55:25.843168 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.843139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8tr\" (UniqueName: \"kubernetes.io/projected/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-kube-api-access-hd8tr\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.843305 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.843180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-serving-cert\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.843305 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.843207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-config\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.843305 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.843258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-oauth-serving-cert\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.843305 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.843300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-oauth-config\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.843440 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.843337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-service-ca\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.944117 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.944090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-config\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.944194 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.944124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-oauth-serving-cert\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.944194 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.944155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-oauth-config\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.944263 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.944203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-service-ca\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.944263 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.944238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8tr\" (UniqueName: \"kubernetes.io/projected/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-kube-api-access-hd8tr\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.944324 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.944277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-serving-cert\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.944914 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.944888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-oauth-serving-cert\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.944914 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.944906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-config\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.945091 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.944893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-service-ca\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.946646 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.946616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-oauth-config\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.946757 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.946700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-serving-cert\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:25.951527 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:25.951508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8tr\" (UniqueName: \"kubernetes.io/projected/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-kube-api-access-hd8tr\") pod \"console-5478c48966-jtvh2\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") " pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:26.056630 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:26.056552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:26.324030 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:26.323949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5478c48966-jtvh2"]
Sep 15 18:55:26.327910 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:55:26.327881    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5ae1fa_0925_4ff0_8bd7_5e814e74cbce.slice/crio-a4d02dcac5209665052a85ceddd5f5f2441f8e8488f4e1b7984feae15e3907d4 WatchSource:0}: Error finding container a4d02dcac5209665052a85ceddd5f5f2441f8e8488f4e1b7984feae15e3907d4: Status 404 returned error can't find the container with id a4d02dcac5209665052a85ceddd5f5f2441f8e8488f4e1b7984feae15e3907d4
Sep 15 18:55:27.017137 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:27.017098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5478c48966-jtvh2" event={"ID":"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce","Type":"ContainerStarted","Data":"a4d02dcac5209665052a85ceddd5f5f2441f8e8488f4e1b7984feae15e3907d4"}
Sep 15 18:55:27.403209 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:27.403168    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"]
Sep 15 18:55:27.406853 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:27.406822    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"
Sep 15 18:55:27.409417 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:27.408891    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 15 18:55:27.409417 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:27.409009    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lpl8n\""
Sep 15 18:55:27.411436 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:27.411410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"]
Sep 15 18:55:27.555545 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:27.555494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1cec380a-fe2b-4919-b539-de206b422c84-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-2wcdq\" (UID: \"1cec380a-fe2b-4919-b539-de206b422c84\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"
Sep 15 18:55:27.657140 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:27.657039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1cec380a-fe2b-4919-b539-de206b422c84-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-2wcdq\" (UID: \"1cec380a-fe2b-4919-b539-de206b422c84\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"
Sep 15 18:55:27.657305 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:27.657175    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 15 18:55:27.657305 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:27.657268    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cec380a-fe2b-4919-b539-de206b422c84-tls-certificates podName:1cec380a-fe2b-4919-b539-de206b422c84 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:28.157246021 +0000 UTC m=+278.699341536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1cec380a-fe2b-4919-b539-de206b422c84-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-2wcdq" (UID: "1cec380a-fe2b-4919-b539-de206b422c84") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 15 18:55:28.161657 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:28.161612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1cec380a-fe2b-4919-b539-de206b422c84-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-2wcdq\" (UID: \"1cec380a-fe2b-4919-b539-de206b422c84\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"
Sep 15 18:55:28.164383 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:28.164351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1cec380a-fe2b-4919-b539-de206b422c84-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-2wcdq\" (UID: \"1cec380a-fe2b-4919-b539-de206b422c84\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"
Sep 15 18:55:28.318462 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:28.318422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"
Sep 15 18:55:28.594143 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:28.594058    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"]
Sep 15 18:55:28.597555 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:55:28.597525    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cec380a_fe2b_4919_b539_de206b422c84.slice/crio-5a701f241c38c77c6e3b1d9e5f3a63a2f9d0b8db0453c0865be08112e4842ae9 WatchSource:0}: Error finding container 5a701f241c38c77c6e3b1d9e5f3a63a2f9d0b8db0453c0865be08112e4842ae9: Status 404 returned error can't find the container with id 5a701f241c38c77c6e3b1d9e5f3a63a2f9d0b8db0453c0865be08112e4842ae9
Sep 15 18:55:29.024904 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:29.024859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq" event={"ID":"1cec380a-fe2b-4919-b539-de206b422c84","Type":"ContainerStarted","Data":"5a701f241c38c77c6e3b1d9e5f3a63a2f9d0b8db0453c0865be08112e4842ae9"}
Sep 15 18:55:29.975704 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:29.975667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:55:29.978259 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:29.978235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ac0fda6-2405-4bcf-a74b-612af3c2268c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hn444\" (UID: \"3ac0fda6-2405-4bcf-a74b-612af3c2268c\") " pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:55:29.995697 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:29.995672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hn444"
Sep 15 18:55:30.031560 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:30.031507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5478c48966-jtvh2" event={"ID":"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce","Type":"ContainerStarted","Data":"34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01"}
Sep 15 18:55:30.048007 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:30.047961    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5478c48966-jtvh2" podStartSLOduration=1.5079518159999998 podStartE2EDuration="5.047944397s" totalImagesPullingTime="3.539992581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:30.046757454 +0000 UTC m=+280.588852984" watchObservedRunningTime="2026-09-15 18:55:30.047944397 +0000 UTC m=+280.590039926"
Sep 15 18:55:30.277653 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:30.277614    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hn444"]
Sep 15 18:55:30.281155 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:55:30.281111    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac0fda6_2405_4bcf_a74b_612af3c2268c.slice/crio-a6fbfa45917e61f15163cfa2bce3cc7a1d1752ed5cd8b75ddfdaffadfaf03008 WatchSource:0}: Error finding container a6fbfa45917e61f15163cfa2bce3cc7a1d1752ed5cd8b75ddfdaffadfaf03008: Status 404 returned error can't find the container with id a6fbfa45917e61f15163cfa2bce3cc7a1d1752ed5cd8b75ddfdaffadfaf03008
Sep 15 18:55:30.679629 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:30.679597    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bngct" podUID="05d53002-a924-4ea1-a656-da4e6ba1e6db"
Sep 15 18:55:31.036031 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:31.035993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq" event={"ID":"1cec380a-fe2b-4919-b539-de206b422c84","Type":"ContainerStarted","Data":"ca3dfbd0feb3ba496b567ae9c8a5dd0b9dcd2df48b5e34f7e9886165626dd97c"}
Sep 15 18:55:31.036460 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:31.036179    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"
Sep 15 18:55:31.037557 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:31.037535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bngct"
Sep 15 18:55:31.037685 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:31.037573    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hn444" event={"ID":"3ac0fda6-2405-4bcf-a74b-612af3c2268c","Type":"ContainerStarted","Data":"830a6db91906d7d43aee3d5a5b1448988066da742a5889fe4664b114de75fbe0"}
Sep 15 18:55:31.037685 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:31.037628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hn444" event={"ID":"3ac0fda6-2405-4bcf-a74b-612af3c2268c","Type":"ContainerStarted","Data":"a6fbfa45917e61f15163cfa2bce3cc7a1d1752ed5cd8b75ddfdaffadfaf03008"}
Sep 15 18:55:31.040919 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:31.040899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq"
Sep 15 18:55:31.062541 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:31.062489    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-2wcdq" podStartSLOduration=2.075098251 podStartE2EDuration="4.062472238s" totalImagesPullingTime="1.987373987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:31.049357822 +0000 UTC m=+281.591453352" watchObservedRunningTime="2026-09-15 18:55:31.062472238 +0000 UTC m=+281.604567764"
Sep 15 18:55:32.043616 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:32.043571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hn444" event={"ID":"3ac0fda6-2405-4bcf-a74b-612af3c2268c","Type":"ContainerStarted","Data":"5b42239eecd4219844dacde92fac9915e1e4156045b4a5d501b1a7c97bc0797d"}
Sep 15 18:55:33.051448 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:33.051362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hn444" event={"ID":"3ac0fda6-2405-4bcf-a74b-612af3c2268c","Type":"ContainerStarted","Data":"3ce74115b45b3f364cca23221a9d10a0c427f69ce15f77e1eb8d6f8c4ccde8c9"}
Sep 15 18:55:33.069643 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:33.069595    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hn444" podStartSLOduration=32.811360081 podStartE2EDuration="35.069581834s" totalImagesPullingTime="2.258221753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:54:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:33.068637758 +0000 UTC m=+283.610733299" watchObservedRunningTime="2026-09-15 18:55:33.069581834 +0000 UTC m=+283.611677363"
Sep 15 18:55:34.316438 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:34.316398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:55:34.318638 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:34.318620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05d53002-a924-4ea1-a656-da4e6ba1e6db-metrics-tls\") pod \"dns-default-bngct\" (UID: \"05d53002-a924-4ea1-a656-da4e6ba1e6db\") " pod="openshift-dns/dns-default-bngct"
Sep 15 18:55:34.341023 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:34.340992    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hj774\""
Sep 15 18:55:34.349138 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:34.349119    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bngct"
Sep 15 18:55:34.527303 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:34.527262    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" podUID="5ce5c596-e80c-4907-bef1-9d0c360bdfcd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 18:55:34.614564 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:34.614484    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bngct"]
Sep 15 18:55:34.617190 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:55:34.617162    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d53002_a924_4ea1_a656_da4e6ba1e6db.slice/crio-4a2dce1c777236ec995eeb2b06630d7c90b58f42c2f0189e60efbb31acdc68cc WatchSource:0}: Error finding container 4a2dce1c777236ec995eeb2b06630d7c90b58f42c2f0189e60efbb31acdc68cc: Status 404 returned error can't find the container with id 4a2dce1c777236ec995eeb2b06630d7c90b58f42c2f0189e60efbb31acdc68cc
Sep 15 18:55:35.059637 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.059601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bngct" event={"ID":"05d53002-a924-4ea1-a656-da4e6ba1e6db","Type":"ContainerStarted","Data":"4a2dce1c777236ec995eeb2b06630d7c90b58f42c2f0189e60efbb31acdc68cc"}
Sep 15 18:55:35.799377 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.799339    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7r5v6"]
Sep 15 18:55:35.803049 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.803022    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:35.805423 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.805393    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 18:55:35.805535 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.805487    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 18:55:35.806395 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.806369    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xf85x\""
Sep 15 18:55:35.806505 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.806425    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 18:55:35.806572 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.806521    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 18:55:35.806649 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.806615    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 18:55:35.806708 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.806696    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 18:55:35.931308 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.931265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-tls\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:35.931481 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.931328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef6be42b-32b1-4166-bae4-ea69002cb587-root\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:35.931481 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.931394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvwcw\" (UniqueName: \"kubernetes.io/projected/ef6be42b-32b1-4166-bae4-ea69002cb587-kube-api-access-qvwcw\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:35.931481 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.931443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef6be42b-32b1-4166-bae4-ea69002cb587-sys\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:35.931622 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.931535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-wtmp\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:35.931622 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.931575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-accelerators-collector-config\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:35.931692 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.931632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:35.931692 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.931677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-textfile\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:35.931772 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:35.931715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef6be42b-32b1-4166-bae4-ea69002cb587-metrics-client-ca\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.032842 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.032790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-textfile\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033006 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.032855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef6be42b-32b1-4166-bae4-ea69002cb587-metrics-client-ca\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033006 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.032888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-tls\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033006 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.032938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef6be42b-32b1-4166-bae4-ea69002cb587-root\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033006 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.032966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvwcw\" (UniqueName: \"kubernetes.io/projected/ef6be42b-32b1-4166-bae4-ea69002cb587-kube-api-access-qvwcw\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033006 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.032999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef6be42b-32b1-4166-bae4-ea69002cb587-sys\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033274 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.033056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-wtmp\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033274 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:36.033071    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 18:55:36.033274 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.033095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-accelerators-collector-config\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033274 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:36.033137    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-tls podName:ef6be42b-32b1-4166-bae4-ea69002cb587 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:36.533116766 +0000 UTC m=+287.075212292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-tls") pod "node-exporter-7r5v6" (UID: "ef6be42b-32b1-4166-bae4-ea69002cb587") : secret "node-exporter-tls" not found
Sep 15 18:55:36.033274 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.033168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033274 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.033187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef6be42b-32b1-4166-bae4-ea69002cb587-root\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033274 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.033216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-textfile\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033616 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.033345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-wtmp\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033616 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.033345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef6be42b-32b1-4166-bae4-ea69002cb587-sys\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033616 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.033531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-accelerators-collector-config\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.033616 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.033551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef6be42b-32b1-4166-bae4-ea69002cb587-metrics-client-ca\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.035793 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.035768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.041714 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.041690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvwcw\" (UniqueName: \"kubernetes.io/projected/ef6be42b-32b1-4166-bae4-ea69002cb587-kube-api-access-qvwcw\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.057577 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.057517    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:36.057685 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.057581    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:55:36.059249 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.059225    2578 patch_prober.go:28] interesting pod/console-5478c48966-jtvh2 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.11:8443/health\": dial tcp 10.134.0.11:8443: connect: connection refused" start-of-body=
Sep 15 18:55:36.059373 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.059268    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5478c48966-jtvh2" podUID="df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" containerName="console" probeResult="failure" output="Get \"https://10.134.0.11:8443/health\": dial tcp 10.134.0.11:8443: connect: connection refused"
Sep 15 18:55:36.537109 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.537062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-tls\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.539321 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.539288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef6be42b-32b1-4166-bae4-ea69002cb587-node-exporter-tls\") pod \"node-exporter-7r5v6\" (UID: \"ef6be42b-32b1-4166-bae4-ea69002cb587\") " pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.714225 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:36.714189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7r5v6"
Sep 15 18:55:36.724230 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:55:36.724207    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6be42b_32b1_4166_bae4_ea69002cb587.slice/crio-cc12e0b15aab255c9c0e4bc62e037db9747fcfbf37ecc34f295efcec9bf500a1 WatchSource:0}: Error finding container cc12e0b15aab255c9c0e4bc62e037db9747fcfbf37ecc34f295efcec9bf500a1: Status 404 returned error can't find the container with id cc12e0b15aab255c9c0e4bc62e037db9747fcfbf37ecc34f295efcec9bf500a1
Sep 15 18:55:37.068104 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:37.068070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r5v6" event={"ID":"ef6be42b-32b1-4166-bae4-ea69002cb587","Type":"ContainerStarted","Data":"cc12e0b15aab255c9c0e4bc62e037db9747fcfbf37ecc34f295efcec9bf500a1"}
Sep 15 18:55:37.069902 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:37.069881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bngct" event={"ID":"05d53002-a924-4ea1-a656-da4e6ba1e6db","Type":"ContainerStarted","Data":"f04c715a9c3d884d64d17ec99fce5e8112cc84d05c07563d7c22ba0a2cdd8011"}
Sep 15 18:55:37.069959 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:37.069926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bngct" event={"ID":"05d53002-a924-4ea1-a656-da4e6ba1e6db","Type":"ContainerStarted","Data":"bf0c9f251059c99a85793dca6f585bd1388c1ebb2a8905641aa19a311507c95c"}
Sep 15 18:55:37.070054 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:37.070038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bngct"
Sep 15 18:55:37.086526 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:37.086481    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bngct" podStartSLOduration=251.567757875 podStartE2EDuration="4m13.086466743s" totalImagesPullingTime="1.518708868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:37.085926245 +0000 UTC m=+287.628021788" watchObservedRunningTime="2026-09-15 18:55:37.086466743 +0000 UTC m=+287.628562266"
Sep 15 18:55:38.074738 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.074712    2578 generic.go:356] "Generic (PLEG): container finished" podID="ef6be42b-32b1-4166-bae4-ea69002cb587" containerID="dcc5ce4b40eaf1df37f142d59c676731e604f7511a7c6d763820b05c825e6148" exitCode=0
Sep 15 18:55:38.075095 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.074819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r5v6" event={"ID":"ef6be42b-32b1-4166-bae4-ea69002cb587","Type":"ContainerDied","Data":"dcc5ce4b40eaf1df37f142d59c676731e604f7511a7c6d763820b05c825e6148"}
Sep 15 18:55:38.776087 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.776055    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"]
Sep 15 18:55:38.779340 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.779324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:38.781660 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.781637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 18:55:38.781778 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.781637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 18:55:38.781976 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.781951    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 18:55:38.782106 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.782089    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 18:55:38.782162 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.782099    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zkhzs\""
Sep 15 18:55:38.782220 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.782165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8h2f76sjirbb7\""
Sep 15 18:55:38.782220 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.782179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 18:55:38.790306 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.790285    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"]
Sep 15 18:55:38.957140 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.957112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:38.957140 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.957142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-metrics-client-ca\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:38.957344 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.957170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-grpc-tls\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:38.957344 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.957189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:38.957344 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.957263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6gl5\" (UniqueName: \"kubernetes.io/projected/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-kube-api-access-r6gl5\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:38.957344 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.957318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-tls\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:38.957517 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.957350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:38.957517 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:38.957398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.058512 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.058441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-tls\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.058512 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.058483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.058512 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.058513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.058720 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.058530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.058720 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.058547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-metrics-client-ca\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.058720 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.058572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-grpc-tls\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.058720 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.058593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.058720 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.058623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6gl5\" (UniqueName: \"kubernetes.io/projected/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-kube-api-access-r6gl5\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.059422 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.059336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-metrics-client-ca\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.061212 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.061167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.061212 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.061200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.061400 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.061382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-grpc-tls\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.061469 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.061452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.061525 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.061489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-tls\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.061582 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.061544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.066179 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.066160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6gl5\" (UniqueName: \"kubernetes.io/projected/b1c57538-53ad-4a20-aaa9-21ce2a5e70c4-kube-api-access-r6gl5\") pod \"thanos-querier-57875dc5d5-s4gxw\" (UID: \"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4\") " pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.079445 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.079424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r5v6" event={"ID":"ef6be42b-32b1-4166-bae4-ea69002cb587","Type":"ContainerStarted","Data":"e24436624538059869568124ecac887290131b7b0c4791f8536ffa416a0168d8"}
Sep 15 18:55:39.079791 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.079477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7r5v6" event={"ID":"ef6be42b-32b1-4166-bae4-ea69002cb587","Type":"ContainerStarted","Data":"e90a4a216d07926ff9b0f0bea91291171bcef9c14617b77c38423d4c4cc0adb2"}
Sep 15 18:55:39.088482 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.088461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:39.097154 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.097117    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7r5v6" podStartSLOduration=2.848922686 podStartE2EDuration="4.097104589s" totalImagesPullingTime="1.187830541s" totalInitContainerRuntime="60.351362ms" isStatefulPod=true podCreationTimestamp="2026-09-15 18:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:39.096362647 +0000 UTC m=+289.638458177" watchObservedRunningTime="2026-09-15 18:55:39.097104589 +0000 UTC m=+289.639200108"
Sep 15 18:55:39.361414 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:39.361388    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"]
Sep 15 18:55:39.363136 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:55:39.363109    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c57538_53ad_4a20_aaa9_21ce2a5e70c4.slice/crio-901c8d4ef08cdf6440c4443b29cf622e4038fb4ffa20f6b4a9df37a1d5d6627e WatchSource:0}: Error finding container 901c8d4ef08cdf6440c4443b29cf622e4038fb4ffa20f6b4a9df37a1d5d6627e: Status 404 returned error can't find the container with id 901c8d4ef08cdf6440c4443b29cf622e4038fb4ffa20f6b4a9df37a1d5d6627e
Sep 15 18:55:40.084480 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:40.084445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw" event={"ID":"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4","Type":"ContainerStarted","Data":"901c8d4ef08cdf6440c4443b29cf622e4038fb4ffa20f6b4a9df37a1d5d6627e"}
Sep 15 18:55:40.743322 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:40.743284    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5478c48966-jtvh2"]
Sep 15 18:55:42.097451 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:42.097416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw" event={"ID":"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4","Type":"ContainerStarted","Data":"ca707721b2a6dccf8c3e2feededd36f1f08373aa4055e3b2582667e9d1af237f"}
Sep 15 18:55:42.097927 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:42.097478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw" event={"ID":"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4","Type":"ContainerStarted","Data":"9c604a80298813ad8c1b5a7a2a00d7adf62c3f81e0a5f006853511929da53816"}
Sep 15 18:55:42.097927 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:42.097503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw" event={"ID":"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4","Type":"ContainerStarted","Data":"a51403bf0c9a7f32aff21ebd06ce6dbddbd77e8cf4381ecf17ff945c95b9f4b1"}
Sep 15 18:55:43.096936 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:55:43.096898    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hktjq" podUID="eb8d5cee-7842-4f44-9079-94434a03ccff"
Sep 15 18:55:43.101271 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:43.101245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw" event={"ID":"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4","Type":"ContainerStarted","Data":"9afc575c7c44b97b1582cee33ada176d49aa8758189a0efce198da75233e2153"}
Sep 15 18:55:43.101652 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:43.101303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw" event={"ID":"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4","Type":"ContainerStarted","Data":"a8892fc155957940b4c52d8374705648f2586d9365fd0d11c955812b69f3894c"}
Sep 15 18:55:43.101652 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:43.101329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw" event={"ID":"b1c57538-53ad-4a20-aaa9-21ce2a5e70c4","Type":"ContainerStarted","Data":"85e5cd5ec3579658ca9b65c4e447233365fbfa46103c2bb5913edc06426c7391"}
Sep 15 18:55:43.101652 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:43.101404    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:43.121189 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:43.121144    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw" podStartSLOduration=2.353142869 podStartE2EDuration="5.1211286s" totalImagesPullingTime="2.767985731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:43.120655853 +0000 UTC m=+293.662751381" watchObservedRunningTime="2026-09-15 18:55:43.1211286 +0000 UTC m=+293.663224129"
Sep 15 18:55:44.526854 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:44.526793    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" podUID="5ce5c596-e80c-4907-bef1-9d0c360bdfcd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 18:55:47.077136 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:47.077104    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bngct"
Sep 15 18:55:49.110067 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:49.110039    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57875dc5d5-s4gxw"
Sep 15 18:55:49.898772 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:49.898747    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 18:55:54.526614 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:54.526577    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" podUID="5ce5c596-e80c-4907-bef1-9d0c360bdfcd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 18:55:54.529031 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:54.526648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h"
Sep 15 18:55:54.529031 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:54.527084    2578 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"75108dee8d6c932ad4ff4a545ad22c536b3ca8d89636be543e498273c981c763"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 15 18:55:54.529031 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:54.527135    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" podUID="5ce5c596-e80c-4907-bef1-9d0c360bdfcd" containerName="service-proxy" containerID="cri-o://75108dee8d6c932ad4ff4a545ad22c536b3ca8d89636be543e498273c981c763" gracePeriod=30
Sep 15 18:55:55.141579 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:55.141540    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ce5c596-e80c-4907-bef1-9d0c360bdfcd" containerID="75108dee8d6c932ad4ff4a545ad22c536b3ca8d89636be543e498273c981c763" exitCode=2
Sep 15 18:55:55.141741 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:55.141601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" event={"ID":"5ce5c596-e80c-4907-bef1-9d0c360bdfcd","Type":"ContainerDied","Data":"75108dee8d6c932ad4ff4a545ad22c536b3ca8d89636be543e498273c981c763"}
Sep 15 18:55:55.141741 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:55.141644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77cf6bc965-r5w5h" event={"ID":"5ce5c596-e80c-4907-bef1-9d0c360bdfcd","Type":"ContainerStarted","Data":"bb80b6e9701ab1a30d81502524b52722499c84b2a2dc5d3f7b0f57f577998e5a"}
Sep 15 18:55:57.093043 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:55:57.093013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:56:05.761714 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:05.761659    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5478c48966-jtvh2" podUID="df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" containerName="console" containerID="cri-o://34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01" gracePeriod=15
Sep 15 18:56:05.996522 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:05.996495    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:56:06.046841 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.046754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd8tr\" (UniqueName: \"kubernetes.io/projected/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-kube-api-access-hd8tr\") pod \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") "
Sep 15 18:56:06.046841 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.046815    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-oauth-serving-cert\") pod \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") "
Sep 15 18:56:06.047027 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.046845    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-serving-cert\") pod \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") "
Sep 15 18:56:06.047027 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.046902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-service-ca\") pod \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") "
Sep 15 18:56:06.047027 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.046928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-oauth-config\") pod \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") "
Sep 15 18:56:06.047027 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.046975    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-config\") pod \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\" (UID: \"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce\") "
Sep 15 18:56:06.047304 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.047277    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" (UID: "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:56:06.047429 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.047382    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-config" (OuterVolumeSpecName: "console-config") pod "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" (UID: "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:56:06.047496 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.047450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-service-ca" (OuterVolumeSpecName: "service-ca") pod "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" (UID: "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:56:06.049059 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.049036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" (UID: "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:56:06.049150 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.049112    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-kube-api-access-hd8tr" (OuterVolumeSpecName: "kube-api-access-hd8tr") pod "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" (UID: "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce"). InnerVolumeSpecName "kube-api-access-hd8tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:56:06.049260 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.049238    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" (UID: "df5ae1fa-0925-4ff0-8bd7-5e814e74cbce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:56:06.147847 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.147819    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-config\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:56:06.147939 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.147848    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd8tr\" (UniqueName: \"kubernetes.io/projected/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-kube-api-access-hd8tr\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:56:06.147939 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.147863    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-oauth-serving-cert\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:56:06.147939 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.147876    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-serving-cert\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:56:06.147939 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.147890    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:56:06.147939 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.147903    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce-console-oauth-config\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:56:06.178559 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.178535    2578 generic.go:356] "Generic (PLEG): container finished" podID="df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" containerID="34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01" exitCode=0
Sep 15 18:56:06.178649 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.178569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5478c48966-jtvh2" event={"ID":"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce","Type":"ContainerDied","Data":"34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01"}
Sep 15 18:56:06.178649 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.178585    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5478c48966-jtvh2"
Sep 15 18:56:06.178649 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.178631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5478c48966-jtvh2" event={"ID":"df5ae1fa-0925-4ff0-8bd7-5e814e74cbce","Type":"ContainerDied","Data":"a4d02dcac5209665052a85ceddd5f5f2441f8e8488f4e1b7984feae15e3907d4"}
Sep 15 18:56:06.178757 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.178650    2578 scope.go:117] "RemoveContainer" containerID="34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01"
Sep 15 18:56:06.186294 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.186277    2578 scope.go:117] "RemoveContainer" containerID="34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01"
Sep 15 18:56:06.186553 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:56:06.186535    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01\": container with ID starting with 34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01 not found: ID does not exist" containerID="34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01"
Sep 15 18:56:06.186607 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.186561    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01"} err="failed to get container status \"34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01\": rpc error: code = NotFound desc = could not find container \"34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01\": container with ID starting with 34f8a2267476ea53b06281a80a967dbe1b21b67e3c6e993ea4ae279444bcee01 not found: ID does not exist"
Sep 15 18:56:06.217607 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.217575    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5478c48966-jtvh2"]
Sep 15 18:56:06.226762 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:06.226736    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5478c48966-jtvh2"]
Sep 15 18:56:08.097222 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:56:08.097190    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" path="/var/lib/kubelet/pods/df5ae1fa-0925-4ff0-8bd7-5e814e74cbce/volumes"
Sep 15 18:57:03.858287 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:03.858242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:57:03.860496 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:03.860465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb8d5cee-7842-4f44-9079-94434a03ccff-metrics-certs\") pod \"network-metrics-daemon-hktjq\" (UID: \"eb8d5cee-7842-4f44-9079-94434a03ccff\") " pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:57:03.996715 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:03.996683    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bc8c\""
Sep 15 18:57:04.004824 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.004791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hktjq"
Sep 15 18:57:04.277771 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.277718    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hktjq"]
Sep 15 18:57:04.283751 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:57:04.283722    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8d5cee_7842_4f44_9079_94434a03ccff.slice/crio-412ebfb5c57277f72859881f53f07c4727a1792d4d5f25b7d5c28500bb7e7a22 WatchSource:0}: Error finding container 412ebfb5c57277f72859881f53f07c4727a1792d4d5f25b7d5c28500bb7e7a22: Status 404 returned error can't find the container with id 412ebfb5c57277f72859881f53f07c4727a1792d4d5f25b7d5c28500bb7e7a22
Sep 15 18:57:04.286174 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.286154    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 18:57:04.373095 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.373059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hktjq" event={"ID":"eb8d5cee-7842-4f44-9079-94434a03ccff","Type":"ContainerStarted","Data":"412ebfb5c57277f72859881f53f07c4727a1792d4d5f25b7d5c28500bb7e7a22"}
Sep 15 18:57:04.599973 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.599898    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d485c4bd-c4zds"]
Sep 15 18:57:04.600150 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.600138    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" containerName="console"
Sep 15 18:57:04.600191 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.600152    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" containerName="console"
Sep 15 18:57:04.600224 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.600192    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="df5ae1fa-0925-4ff0-8bd7-5e814e74cbce" containerName="console"
Sep 15 18:57:04.604031 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.604016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.608148 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.608122    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 18:57:04.608272 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.608208    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 18:57:04.610350 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.610280    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 18:57:04.610350 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.610308    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 18:57:04.610863 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.610398    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 18:57:04.610863 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.610487    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 18:57:04.611084 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.611064    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kbhdv\""
Sep 15 18:57:04.611288 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.611272    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 18:57:04.615161 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.615144    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 18:57:04.621977 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.621951    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d485c4bd-c4zds"]
Sep 15 18:57:04.765097 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.765062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-oauth-config\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.765296 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.765108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8fc4\" (UniqueName: \"kubernetes.io/projected/cee72ebd-bf05-45bf-b816-a119c78764e4-kube-api-access-z8fc4\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.765296 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.765128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-oauth-serving-cert\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.765296 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.765228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-trusted-ca-bundle\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.765296 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.765267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-console-config\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.765296 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.765288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-serving-cert\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.765473 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.765304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-service-ca\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.866328 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.866247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-trusted-ca-bundle\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.866328 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.866290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-console-config\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.866328 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.866317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-serving-cert\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.866909 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.866338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-service-ca\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.866909 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.866371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-oauth-config\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.866909 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.866404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fc4\" (UniqueName: \"kubernetes.io/projected/cee72ebd-bf05-45bf-b816-a119c78764e4-kube-api-access-z8fc4\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.866909 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.866437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-oauth-serving-cert\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.867151 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.867127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-console-config\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.867204 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.867127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-service-ca\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.867379 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.867353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-trusted-ca-bundle\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.867730 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.867705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-oauth-serving-cert\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.869313 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.869292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-serving-cert\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.869948 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.869926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-oauth-config\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.875239 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.875218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8fc4\" (UniqueName: \"kubernetes.io/projected/cee72ebd-bf05-45bf-b816-a119c78764e4-kube-api-access-z8fc4\") pod \"console-6d485c4bd-c4zds\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") " pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:04.913244 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:04.913208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:05.197623 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:05.197591    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d485c4bd-c4zds"]
Sep 15 18:57:05.268950 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:57:05.268917    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee72ebd_bf05_45bf_b816_a119c78764e4.slice/crio-839dc04270b1fb353e82da78d3ae400bedf8857556987f27ce45f6b128440f89 WatchSource:0}: Error finding container 839dc04270b1fb353e82da78d3ae400bedf8857556987f27ce45f6b128440f89: Status 404 returned error can't find the container with id 839dc04270b1fb353e82da78d3ae400bedf8857556987f27ce45f6b128440f89
Sep 15 18:57:05.377508 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:05.377476    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d485c4bd-c4zds" event={"ID":"cee72ebd-bf05-45bf-b816-a119c78764e4","Type":"ContainerStarted","Data":"c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443"}
Sep 15 18:57:05.377674 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:05.377524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d485c4bd-c4zds" event={"ID":"cee72ebd-bf05-45bf-b816-a119c78764e4","Type":"ContainerStarted","Data":"839dc04270b1fb353e82da78d3ae400bedf8857556987f27ce45f6b128440f89"}
Sep 15 18:57:05.419674 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:05.419620    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d485c4bd-c4zds" podStartSLOduration=1.419599501 podStartE2EDuration="1.419599501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:05.419401531 +0000 UTC m=+375.961497062" watchObservedRunningTime="2026-09-15 18:57:05.419599501 +0000 UTC m=+375.961695034"
Sep 15 18:57:06.382759 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:06.382724    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hktjq" event={"ID":"eb8d5cee-7842-4f44-9079-94434a03ccff","Type":"ContainerStarted","Data":"587cddd35f9ec07a98f2c6d1c648b0001e40102e628d55ca3072838bd07b9acf"}
Sep 15 18:57:06.383150 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:06.382775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hktjq" event={"ID":"eb8d5cee-7842-4f44-9079-94434a03ccff","Type":"ContainerStarted","Data":"27b7a8f3f1e245618ef2027c3568a2a1356591309e1e3ea92843a37a1e7f6ba4"}
Sep 15 18:57:06.399064 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:06.399020    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hktjq" podStartSLOduration=375.367010797 podStartE2EDuration="6m16.399005024s" totalImagesPullingTime="1.031994227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:06.39836562 +0000 UTC m=+376.940461158" watchObservedRunningTime="2026-09-15 18:57:06.399005024 +0000 UTC m=+376.941100553"
Sep 15 18:57:14.914003 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:14.913955    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:14.914509 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:14.914073    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:14.918438 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:14.918417    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:15.421546 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.421512    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:15.595619 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.595589    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b7dbf4dc-2pkvd"]
Sep 15 18:57:15.599734 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.599713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.610512 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.610486    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7dbf4dc-2pkvd"]
Sep 15 18:57:15.747365 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.747295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8m66\" (UniqueName: \"kubernetes.io/projected/0b274ca5-2403-44cf-bcd3-adcd19d5b448-kube-api-access-n8m66\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.747365 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.747329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-oauth-config\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.747365 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.747356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-config\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.747567 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.747401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-serving-cert\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.747567 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.747420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-service-ca\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.747567 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.747438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-trusted-ca-bundle\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.747567 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.747468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-oauth-serving-cert\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.847769 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.847742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-oauth-serving-cert\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.847880 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.847776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8m66\" (UniqueName: \"kubernetes.io/projected/0b274ca5-2403-44cf-bcd3-adcd19d5b448-kube-api-access-n8m66\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.847880 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.847814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-oauth-config\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.847880 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.847837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-config\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.847880 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.847862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-serving-cert\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.848077 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.847882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-service-ca\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.848077 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.847899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-trusted-ca-bundle\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.848663 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.848636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-oauth-serving-cert\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.848663 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.848656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-config\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.848775 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.848679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-service-ca\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.848775 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.848679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-trusted-ca-bundle\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.850166 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.850142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-oauth-config\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.850260 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.850242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-serving-cert\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.855815 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.855780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8m66\" (UniqueName: \"kubernetes.io/projected/0b274ca5-2403-44cf-bcd3-adcd19d5b448-kube-api-access-n8m66\") pod \"console-7b7dbf4dc-2pkvd\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") " pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:15.909050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:15.909029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:16.175785 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:16.175757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7dbf4dc-2pkvd"]
Sep 15 18:57:16.178895 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:57:16.178869    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b274ca5_2403_44cf_bcd3_adcd19d5b448.slice/crio-051098f3adea743f83da1fdecbe9f657652e1b4c803a3d8c49f89d4fa4865a3d WatchSource:0}: Error finding container 051098f3adea743f83da1fdecbe9f657652e1b4c803a3d8c49f89d4fa4865a3d: Status 404 returned error can't find the container with id 051098f3adea743f83da1fdecbe9f657652e1b4c803a3d8c49f89d4fa4865a3d
Sep 15 18:57:16.423941 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:16.423896    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7dbf4dc-2pkvd" event={"ID":"0b274ca5-2403-44cf-bcd3-adcd19d5b448","Type":"ContainerStarted","Data":"d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd"}
Sep 15 18:57:16.424101 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:16.423959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7dbf4dc-2pkvd" event={"ID":"0b274ca5-2403-44cf-bcd3-adcd19d5b448","Type":"ContainerStarted","Data":"051098f3adea743f83da1fdecbe9f657652e1b4c803a3d8c49f89d4fa4865a3d"}
Sep 15 18:57:16.441042 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:16.440962    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b7dbf4dc-2pkvd" podStartSLOduration=1.440948069 podStartE2EDuration="1.440948069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:16.439042224 +0000 UTC m=+386.981137752" watchObservedRunningTime="2026-09-15 18:57:16.440948069 +0000 UTC m=+386.983043598"
Sep 15 18:57:25.909299 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:25.909261    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:25.909831 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:25.909327    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:25.914050 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:25.914014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:26.465948 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:26.465923    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 18:57:26.515411 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:26.515374    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d485c4bd-c4zds"]
Sep 15 18:57:51.534021 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.533976    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d485c4bd-c4zds" podUID="cee72ebd-bf05-45bf-b816-a119c78764e4" containerName="console" containerID="cri-o://c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443" gracePeriod=15
Sep 15 18:57:51.763155 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.763130    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:51.785176 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.785111    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-oauth-config\") pod \"cee72ebd-bf05-45bf-b816-a119c78764e4\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") "
Sep 15 18:57:51.785176 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.785164    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8fc4\" (UniqueName: \"kubernetes.io/projected/cee72ebd-bf05-45bf-b816-a119c78764e4-kube-api-access-z8fc4\") pod \"cee72ebd-bf05-45bf-b816-a119c78764e4\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") "
Sep 15 18:57:51.785344 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.785223    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-service-ca\") pod \"cee72ebd-bf05-45bf-b816-a119c78764e4\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") "
Sep 15 18:57:51.785344 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.785253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-serving-cert\") pod \"cee72ebd-bf05-45bf-b816-a119c78764e4\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") "
Sep 15 18:57:51.786246 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.785370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-oauth-serving-cert\") pod \"cee72ebd-bf05-45bf-b816-a119c78764e4\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") "
Sep 15 18:57:51.786246 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.785416    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-trusted-ca-bundle\") pod \"cee72ebd-bf05-45bf-b816-a119c78764e4\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") "
Sep 15 18:57:51.786246 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.785461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-console-config\") pod \"cee72ebd-bf05-45bf-b816-a119c78764e4\" (UID: \"cee72ebd-bf05-45bf-b816-a119c78764e4\") "
Sep 15 18:57:51.786246 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.785591    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-service-ca" (OuterVolumeSpecName: "service-ca") pod "cee72ebd-bf05-45bf-b816-a119c78764e4" (UID: "cee72ebd-bf05-45bf-b816-a119c78764e4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:57:51.786246 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.785688    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:57:51.786246 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.785934    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cee72ebd-bf05-45bf-b816-a119c78764e4" (UID: "cee72ebd-bf05-45bf-b816-a119c78764e4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:57:51.786246 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.786022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cee72ebd-bf05-45bf-b816-a119c78764e4" (UID: "cee72ebd-bf05-45bf-b816-a119c78764e4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:57:51.786644 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.786265    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-console-config" (OuterVolumeSpecName: "console-config") pod "cee72ebd-bf05-45bf-b816-a119c78764e4" (UID: "cee72ebd-bf05-45bf-b816-a119c78764e4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:57:51.787532 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.787506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee72ebd-bf05-45bf-b816-a119c78764e4-kube-api-access-z8fc4" (OuterVolumeSpecName: "kube-api-access-z8fc4") pod "cee72ebd-bf05-45bf-b816-a119c78764e4" (UID: "cee72ebd-bf05-45bf-b816-a119c78764e4"). InnerVolumeSpecName "kube-api-access-z8fc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:57:51.787699 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.787676    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cee72ebd-bf05-45bf-b816-a119c78764e4" (UID: "cee72ebd-bf05-45bf-b816-a119c78764e4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:57:51.787740 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.787693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cee72ebd-bf05-45bf-b816-a119c78764e4" (UID: "cee72ebd-bf05-45bf-b816-a119c78764e4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:57:51.886090 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.886061    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-console-config\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:57:51.886090 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.886084    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-oauth-config\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:57:51.886090 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.886093    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8fc4\" (UniqueName: \"kubernetes.io/projected/cee72ebd-bf05-45bf-b816-a119c78764e4-kube-api-access-z8fc4\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:57:51.886270 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.886104    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cee72ebd-bf05-45bf-b816-a119c78764e4-console-serving-cert\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:57:51.886270 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.886113    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-oauth-serving-cert\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:57:51.886270 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:51.886121    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee72ebd-bf05-45bf-b816-a119c78764e4-trusted-ca-bundle\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 18:57:52.552218 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:52.552182    2578 generic.go:356] "Generic (PLEG): container finished" podID="cee72ebd-bf05-45bf-b816-a119c78764e4" containerID="c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443" exitCode=0
Sep 15 18:57:52.552664 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:52.552225    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d485c4bd-c4zds"
Sep 15 18:57:52.552664 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:52.552230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d485c4bd-c4zds" event={"ID":"cee72ebd-bf05-45bf-b816-a119c78764e4","Type":"ContainerDied","Data":"c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443"}
Sep 15 18:57:52.552664 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:52.552277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d485c4bd-c4zds" event={"ID":"cee72ebd-bf05-45bf-b816-a119c78764e4","Type":"ContainerDied","Data":"839dc04270b1fb353e82da78d3ae400bedf8857556987f27ce45f6b128440f89"}
Sep 15 18:57:52.552664 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:52.552304    2578 scope.go:117] "RemoveContainer" containerID="c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443"
Sep 15 18:57:52.560176 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:52.560158    2578 scope.go:117] "RemoveContainer" containerID="c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443"
Sep 15 18:57:52.560432 ip-10-0-135-221 kubenswrapper[2578]: E0915 18:57:52.560415    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443\": container with ID starting with c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443 not found: ID does not exist" containerID="c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443"
Sep 15 18:57:52.560482 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:52.560438    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443"} err="failed to get container status \"c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443\": rpc error: code = NotFound desc = could not find container \"c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443\": container with ID starting with c9b8643523c57b333759385178f328059f7ce3be7f0f302e6f43249dc5c3b443 not found: ID does not exist"
Sep 15 18:57:52.566499 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:52.566474    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d485c4bd-c4zds"]
Sep 15 18:57:52.568490 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:52.568469    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d485c4bd-c4zds"]
Sep 15 18:57:54.097286 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:57:54.097251    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cee72ebd-bf05-45bf-b816-a119c78764e4" path="/var/lib/kubelet/pods/cee72ebd-bf05-45bf-b816-a119c78764e4/volumes"
Sep 15 18:59:05.598278 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.598245    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v4tbs"]
Sep 15 18:59:05.598685 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.598534    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cee72ebd-bf05-45bf-b816-a119c78764e4" containerName="console"
Sep 15 18:59:05.598685 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.598550    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cee72ebd-bf05-45bf-b816-a119c78764e4" containerName="console"
Sep 15 18:59:05.598685 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.598613    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cee72ebd-bf05-45bf-b816-a119c78764e4" containerName="console"
Sep 15 18:59:05.602403 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.602387    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs"
Sep 15 18:59:05.604748 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.604726    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 18:59:05.605608 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.605589    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-22hfs\""
Sep 15 18:59:05.605664 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.605609    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 18:59:05.617823 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.617781    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v4tbs"]
Sep 15 18:59:05.690705 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.690673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5htqd\" (UniqueName: \"kubernetes.io/projected/9f1e5d00-c4bf-4304-be67-0d3d8457efb3-kube-api-access-5htqd\") pod \"cert-manager-webhook-6486ffd997-v4tbs\" (UID: \"9f1e5d00-c4bf-4304-be67-0d3d8457efb3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs"
Sep 15 18:59:05.690893 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.690712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f1e5d00-c4bf-4304-be67-0d3d8457efb3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v4tbs\" (UID: \"9f1e5d00-c4bf-4304-be67-0d3d8457efb3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs"
Sep 15 18:59:05.791607 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.791573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5htqd\" (UniqueName: \"kubernetes.io/projected/9f1e5d00-c4bf-4304-be67-0d3d8457efb3-kube-api-access-5htqd\") pod \"cert-manager-webhook-6486ffd997-v4tbs\" (UID: \"9f1e5d00-c4bf-4304-be67-0d3d8457efb3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs"
Sep 15 18:59:05.791607 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.791614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f1e5d00-c4bf-4304-be67-0d3d8457efb3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v4tbs\" (UID: \"9f1e5d00-c4bf-4304-be67-0d3d8457efb3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs"
Sep 15 18:59:05.800265 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.800236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f1e5d00-c4bf-4304-be67-0d3d8457efb3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-v4tbs\" (UID: \"9f1e5d00-c4bf-4304-be67-0d3d8457efb3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs"
Sep 15 18:59:05.800391 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.800374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5htqd\" (UniqueName: \"kubernetes.io/projected/9f1e5d00-c4bf-4304-be67-0d3d8457efb3-kube-api-access-5htqd\") pod \"cert-manager-webhook-6486ffd997-v4tbs\" (UID: \"9f1e5d00-c4bf-4304-be67-0d3d8457efb3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs"
Sep 15 18:59:05.910531 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:05.910501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs"
Sep 15 18:59:06.181136 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:06.181053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-v4tbs"]
Sep 15 18:59:06.184756 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:59:06.184727    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f1e5d00_c4bf_4304_be67_0d3d8457efb3.slice/crio-88e4898cddd4de68ff1b13fe9feb52a9c82db20f22f4209ade8a6a7e4f9d8234 WatchSource:0}: Error finding container 88e4898cddd4de68ff1b13fe9feb52a9c82db20f22f4209ade8a6a7e4f9d8234: Status 404 returned error can't find the container with id 88e4898cddd4de68ff1b13fe9feb52a9c82db20f22f4209ade8a6a7e4f9d8234
Sep 15 18:59:06.803499 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:06.803461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs" event={"ID":"9f1e5d00-c4bf-4304-be67-0d3d8457efb3","Type":"ContainerStarted","Data":"88e4898cddd4de68ff1b13fe9feb52a9c82db20f22f4209ade8a6a7e4f9d8234"}
Sep 15 18:59:09.816061 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:09.816025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs" event={"ID":"9f1e5d00-c4bf-4304-be67-0d3d8457efb3","Type":"ContainerStarted","Data":"433d7e99d3bd0480d1e7631dcde6b5a202388fc2571d0444bd21ac941fbb158e"}
Sep 15 18:59:09.816485 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:09.816232    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs"
Sep 15 18:59:09.832038 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:09.831993    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs" podStartSLOduration=1.8774219460000001 podStartE2EDuration="4.831980397s" totalImagesPullingTime="2.954558451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:09.831116497 +0000 UTC m=+500.373212027" watchObservedRunningTime="2026-09-15 18:59:09.831980397 +0000 UTC m=+500.374075926"
Sep 15 18:59:15.821470 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:15.821438    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-v4tbs"
Sep 15 18:59:28.577097 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.577059    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"]
Sep 15 18:59:28.582370 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.582350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.585026 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.585001    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 18:59:28.585789 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.585765    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 18:59:28.585904 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.585765    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 18:59:28.585904 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.585765    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:59:28.585904 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.585834    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dbcbm\""
Sep 15 18:59:28.585904 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.585808    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 18:59:28.593513 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.593489    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"]
Sep 15 18:59:28.754544 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.754517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-cert\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.754701 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.754551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vkr\" (UniqueName: \"kubernetes.io/projected/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-kube-api-access-24vkr\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.754701 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.754568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-metrics-cert\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.754701 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.754686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-manager-config\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.855070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.854993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-manager-config\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.855070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.855034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-cert\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.855070 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.855060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vkr\" (UniqueName: \"kubernetes.io/projected/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-kube-api-access-24vkr\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.855283 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.855102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-metrics-cert\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.855605 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.855585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-manager-config\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.857698 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.857667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-cert\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.857789 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.857673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-metrics-cert\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.862464 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.862443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vkr\" (UniqueName: \"kubernetes.io/projected/1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69-kube-api-access-24vkr\") pod \"lws-controller-manager-5bcb5dcf99-fjjtm\" (UID: \"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69\") " pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:28.891534 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:28.891512    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:29.155289 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:29.155160    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"]
Sep 15 18:59:29.157967 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:59:29.157933    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbf67ad_ec58_489c_a0f4_d6cdfee5cc69.slice/crio-cf944e7bc21aa2199ce9ad193f7c4ea884f2cc935bfa0b1cab99b5630b1657e3 WatchSource:0}: Error finding container cf944e7bc21aa2199ce9ad193f7c4ea884f2cc935bfa0b1cab99b5630b1657e3: Status 404 returned error can't find the container with id cf944e7bc21aa2199ce9ad193f7c4ea884f2cc935bfa0b1cab99b5630b1657e3
Sep 15 18:59:29.885963 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:29.885928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm" event={"ID":"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69","Type":"ContainerStarted","Data":"cf944e7bc21aa2199ce9ad193f7c4ea884f2cc935bfa0b1cab99b5630b1657e3"}
Sep 15 18:59:31.897187 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:31.897104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm" event={"ID":"1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69","Type":"ContainerStarted","Data":"aa1d9783e0dc59b629a731ac5f3ada8627a7a523dd53f7c20a23759887249989"}
Sep 15 18:59:31.897543 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:31.897202    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:31.911461 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:31.911413    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm" podStartSLOduration=1.434676103 podStartE2EDuration="3.911399102s" totalImagesPullingTime="2.476722999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:31.910406949 +0000 UTC m=+522.452502479" watchObservedRunningTime="2026-09-15 18:59:31.911399102 +0000 UTC m=+522.453494630"
Sep 15 18:59:38.676420 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.676383    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"]
Sep 15 18:59:38.679625 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.679609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.681869 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.681844    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 18:59:38.681983 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.681907    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 18:59:38.682165 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.682147    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 18:59:38.682225 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.682167    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 18:59:38.682225 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.682212    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wdkzs\""
Sep 15 18:59:38.695881 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.695860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"]
Sep 15 18:59:38.723473 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.723446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-tmp\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.723583 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.723478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.723583 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.723512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq6x6\" (UniqueName: \"kubernetes.io/projected/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-kube-api-access-cq6x6\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.723583 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.723540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.723717 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.723589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-manifests\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.824465 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.824442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-manifests\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.824568 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.824485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-tmp\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.824568 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.824501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.824568 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.824523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq6x6\" (UniqueName: \"kubernetes.io/projected/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-kube-api-access-cq6x6\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.824568 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.824541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.824978 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.824956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-manifests\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.826830 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.826787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-tmp\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.827216 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.827195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.827257 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.827195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.831643 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.831622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq6x6\" (UniqueName: \"kubernetes.io/projected/9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4-kube-api-access-cq6x6\") pod \"opendatahub-operator-controller-manager-6d9df7495-cl6n5\" (UID: \"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:38.989497 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:38.989436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:39.266528 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:39.266504    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"]
Sep 15 18:59:39.268242 ip-10-0-135-221 kubenswrapper[2578]: W0915 18:59:39.268216    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff8d4ea_79aa_48c2_aaf0_c7cc346bfff4.slice/crio-b0ec4911c6fc987c7b9ea99c2694c0cfc72dc9c872b77f830e3569e0a1c2be32 WatchSource:0}: Error finding container b0ec4911c6fc987c7b9ea99c2694c0cfc72dc9c872b77f830e3569e0a1c2be32: Status 404 returned error can't find the container with id b0ec4911c6fc987c7b9ea99c2694c0cfc72dc9c872b77f830e3569e0a1c2be32
Sep 15 18:59:39.929052 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:39.929008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5" event={"ID":"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4","Type":"ContainerStarted","Data":"b0ec4911c6fc987c7b9ea99c2694c0cfc72dc9c872b77f830e3569e0a1c2be32"}
Sep 15 18:59:42.903256 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:42.903178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5bcb5dcf99-fjjtm"
Sep 15 18:59:42.944583 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:42.944549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5" event={"ID":"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4","Type":"ContainerStarted","Data":"4825c18196e7e50335e6e5af7f9e8d58bccb016f7e73bdb95a1620683e238c43"}
Sep 15 18:59:43.949873 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:43.949837    2578 generic.go:356] "Generic (PLEG): container finished" podID="9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4" containerID="4825c18196e7e50335e6e5af7f9e8d58bccb016f7e73bdb95a1620683e238c43" exitCode=0
Sep 15 18:59:43.950325 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:43.949879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5" event={"ID":"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4","Type":"ContainerDied","Data":"4825c18196e7e50335e6e5af7f9e8d58bccb016f7e73bdb95a1620683e238c43"}
Sep 15 18:59:44.955912 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:44.955875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5" event={"ID":"9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4","Type":"ContainerStarted","Data":"2f071d4a7a64362f30c4547e2ff65b08cdf8a3602c2a19a4cf0d3bea23f7dc47"}
Sep 15 18:59:44.956308 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:44.955985    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 18:59:44.975508 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:44.975465    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5" podStartSLOduration=2.691566349 podStartE2EDuration="6.975451373s" totalImagesPullingTime="3.265455219s" totalInitContainerRuntime="1.018429805s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:44.974901615 +0000 UTC m=+535.516997138" watchObservedRunningTime="2026-09-15 18:59:44.975451373 +0000 UTC m=+535.517546901"
Sep 15 18:59:55.963909 ip-10-0-135-221 kubenswrapper[2578]: I0915 18:59:55.963880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-cl6n5"
Sep 15 19:00:11.602764 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.602726    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-27p29"]
Sep 15 19:00:11.618188 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.618158    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-27p29"]
Sep 15 19:00:11.618353 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.618286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-27p29"
Sep 15 19:00:11.620488 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.620465    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 15 19:00:11.666276 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.666251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psblp\" (UniqueName: \"kubernetes.io/projected/2baaef58-5aa5-465e-8261-e711db65b18b-kube-api-access-psblp\") pod \"model-serving-api-54d89bd5fc-27p29\" (UID: \"2baaef58-5aa5-465e-8261-e711db65b18b\") " pod="opendatahub/model-serving-api-54d89bd5fc-27p29"
Sep 15 19:00:11.666393 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.666288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baaef58-5aa5-465e-8261-e711db65b18b-tls-certs\") pod \"model-serving-api-54d89bd5fc-27p29\" (UID: \"2baaef58-5aa5-465e-8261-e711db65b18b\") " pod="opendatahub/model-serving-api-54d89bd5fc-27p29"
Sep 15 19:00:11.766715 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.766682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psblp\" (UniqueName: \"kubernetes.io/projected/2baaef58-5aa5-465e-8261-e711db65b18b-kube-api-access-psblp\") pod \"model-serving-api-54d89bd5fc-27p29\" (UID: \"2baaef58-5aa5-465e-8261-e711db65b18b\") " pod="opendatahub/model-serving-api-54d89bd5fc-27p29"
Sep 15 19:00:11.766715 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.766720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baaef58-5aa5-465e-8261-e711db65b18b-tls-certs\") pod \"model-serving-api-54d89bd5fc-27p29\" (UID: \"2baaef58-5aa5-465e-8261-e711db65b18b\") " pod="opendatahub/model-serving-api-54d89bd5fc-27p29"
Sep 15 19:00:11.769205 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.769182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baaef58-5aa5-465e-8261-e711db65b18b-tls-certs\") pod \"model-serving-api-54d89bd5fc-27p29\" (UID: \"2baaef58-5aa5-465e-8261-e711db65b18b\") " pod="opendatahub/model-serving-api-54d89bd5fc-27p29"
Sep 15 19:00:11.774955 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.774932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psblp\" (UniqueName: \"kubernetes.io/projected/2baaef58-5aa5-465e-8261-e711db65b18b-kube-api-access-psblp\") pod \"model-serving-api-54d89bd5fc-27p29\" (UID: \"2baaef58-5aa5-465e-8261-e711db65b18b\") " pod="opendatahub/model-serving-api-54d89bd5fc-27p29"
Sep 15 19:00:11.930672 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:11.930628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-27p29"
Sep 15 19:00:12.197698 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:12.197617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-27p29"]
Sep 15 19:00:12.200770 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:00:12.200735    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2baaef58_5aa5_465e_8261_e711db65b18b.slice/crio-3455a669089f1c734b62a60da0a19aa45a27dfdac616aa594fe47dc80d74aefc WatchSource:0}: Error finding container 3455a669089f1c734b62a60da0a19aa45a27dfdac616aa594fe47dc80d74aefc: Status 404 returned error can't find the container with id 3455a669089f1c734b62a60da0a19aa45a27dfdac616aa594fe47dc80d74aefc
Sep 15 19:00:13.062788 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:13.062745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-27p29" event={"ID":"2baaef58-5aa5-465e-8261-e711db65b18b","Type":"ContainerStarted","Data":"3455a669089f1c734b62a60da0a19aa45a27dfdac616aa594fe47dc80d74aefc"}
Sep 15 19:00:15.071894 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:15.071859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-27p29" event={"ID":"2baaef58-5aa5-465e-8261-e711db65b18b","Type":"ContainerStarted","Data":"dd012f79ff5fae230dd01404e8b46844f647c90cf345af4066757e914c56b490"}
Sep 15 19:00:15.072290 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:15.072014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-27p29"
Sep 15 19:00:15.086648 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:15.086600    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-27p29" podStartSLOduration=1.768651192 podStartE2EDuration="4.086588058s" totalImagesPullingTime="2.317936866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:15.085640741 +0000 UTC m=+565.627736270" watchObservedRunningTime="2026-09-15 19:00:15.086588058 +0000 UTC m=+565.628683587"
Sep 15 19:00:18.017699 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.017657    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"]
Sep 15 19:00:18.020586 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.020569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.022923 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.022892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 15 19:00:18.023028 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.022936    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 19:00:18.023236 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.023215    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 15 19:00:18.023352 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.023299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jl9bt\""
Sep 15 19:00:18.023352 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.023254    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 19:00:18.023352 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.023251    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 15 19:00:18.023352 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.023273    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 19:00:18.037548 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.037529    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"]
Sep 15 19:00:18.121822 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.121760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f90aab5f-4144-413c-a9b2-0ce2c73f41da-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.122013 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.121864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.122013 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.121895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f90aab5f-4144-413c-a9b2-0ce2c73f41da-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.122013 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.121928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.122013 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.121965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.122013 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.121988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nqxd\" (UniqueName: \"kubernetes.io/projected/f90aab5f-4144-413c-a9b2-0ce2c73f41da-kube-api-access-8nqxd\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.122013 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.122011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.223291 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.223247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.223291 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.223294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f90aab5f-4144-413c-a9b2-0ce2c73f41da-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.223524 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.223317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.223524 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.223346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.223524 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.223456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nqxd\" (UniqueName: \"kubernetes.io/projected/f90aab5f-4144-413c-a9b2-0ce2c73f41da-kube-api-access-8nqxd\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.223524 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.223495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.223747 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.223547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f90aab5f-4144-413c-a9b2-0ce2c73f41da-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.224154 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.224076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.225705 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.225677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f90aab5f-4144-413c-a9b2-0ce2c73f41da-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.225705 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.225694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.225911 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.225864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.225954 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.225922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f90aab5f-4144-413c-a9b2-0ce2c73f41da-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.237113 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.237084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f90aab5f-4144-413c-a9b2-0ce2c73f41da-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.237204 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.237179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nqxd\" (UniqueName: \"kubernetes.io/projected/f90aab5f-4144-413c-a9b2-0ce2c73f41da-kube-api-access-8nqxd\") pod \"istiod-openshift-gateway-5677ccb49b-8hhr7\" (UID: \"f90aab5f-4144-413c-a9b2-0ce2c73f41da\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.330071 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.329993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:18.606332 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:18.606305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"]
Sep 15 19:00:18.608589 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:00:18.608564    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90aab5f_4144_413c_a9b2_0ce2c73f41da.slice/crio-cecaca12651c51462453b273cbb111c0a6116636e7b042abb2f3a530107e59f8 WatchSource:0}: Error finding container cecaca12651c51462453b273cbb111c0a6116636e7b042abb2f3a530107e59f8: Status 404 returned error can't find the container with id cecaca12651c51462453b273cbb111c0a6116636e7b042abb2f3a530107e59f8
Sep 15 19:00:19.088751 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:19.088712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7" event={"ID":"f90aab5f-4144-413c-a9b2-0ce2c73f41da","Type":"ContainerStarted","Data":"cecaca12651c51462453b273cbb111c0a6116636e7b042abb2f3a530107e59f8"}
Sep 15 19:00:21.048422 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:21.048386    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 19:00:21.048701 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:21.048445    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 19:00:22.104235 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:22.104197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7" event={"ID":"f90aab5f-4144-413c-a9b2-0ce2c73f41da","Type":"ContainerStarted","Data":"c54887182d50ce5adb201b84ecbc81a91064ad6831595c8b50450b2edddac0b0"}
Sep 15 19:00:22.104682 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:22.104398    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:22.106157 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:22.106137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7"
Sep 15 19:00:22.141293 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:22.141248    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8hhr7" podStartSLOduration=2.704010931 podStartE2EDuration="5.141235189s" totalImagesPullingTime="2.437224258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:22.140052457 +0000 UTC m=+572.682147987" watchObservedRunningTime="2026-09-15 19:00:22.141235189 +0000 UTC m=+572.683330718"
Sep 15 19:00:26.079395 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:26.079370    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-27p29"
Sep 15 19:00:40.470906 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:00:40.470864    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b7dbf4dc-2pkvd"]
Sep 15 19:01:05.490127 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.490056    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b7dbf4dc-2pkvd" podUID="0b274ca5-2403-44cf-bcd3-adcd19d5b448" containerName="console" containerID="cri-o://d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd" gracePeriod=15
Sep 15 19:01:05.736068 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.736040    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 19:01:05.765578 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.765493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-oauth-serving-cert\") pod \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") "
Sep 15 19:01:05.765578 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.765542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-trusted-ca-bundle\") pod \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") "
Sep 15 19:01:05.765578 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.765569    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-serving-cert\") pod \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") "
Sep 15 19:01:05.765858 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.765602    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-config\") pod \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") "
Sep 15 19:01:05.765858 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.765643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8m66\" (UniqueName: \"kubernetes.io/projected/0b274ca5-2403-44cf-bcd3-adcd19d5b448-kube-api-access-n8m66\") pod \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") "
Sep 15 19:01:05.765858 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.765674    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-service-ca\") pod \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") "
Sep 15 19:01:05.765858 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.765710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-oauth-config\") pod \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\" (UID: \"0b274ca5-2403-44cf-bcd3-adcd19d5b448\") "
Sep 15 19:01:05.766068 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.765992    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0b274ca5-2403-44cf-bcd3-adcd19d5b448" (UID: "0b274ca5-2403-44cf-bcd3-adcd19d5b448"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:01:05.766068 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.766042    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-config" (OuterVolumeSpecName: "console-config") pod "0b274ca5-2403-44cf-bcd3-adcd19d5b448" (UID: "0b274ca5-2403-44cf-bcd3-adcd19d5b448"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:01:05.766344 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.766304    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0b274ca5-2403-44cf-bcd3-adcd19d5b448" (UID: "0b274ca5-2403-44cf-bcd3-adcd19d5b448"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:01:05.766471 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.766352    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-service-ca" (OuterVolumeSpecName: "service-ca") pod "0b274ca5-2403-44cf-bcd3-adcd19d5b448" (UID: "0b274ca5-2403-44cf-bcd3-adcd19d5b448"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:01:05.767826 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.767785    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b274ca5-2403-44cf-bcd3-adcd19d5b448-kube-api-access-n8m66" (OuterVolumeSpecName: "kube-api-access-n8m66") pod "0b274ca5-2403-44cf-bcd3-adcd19d5b448" (UID: "0b274ca5-2403-44cf-bcd3-adcd19d5b448"). InnerVolumeSpecName "kube-api-access-n8m66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:01:05.767978 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.767954    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0b274ca5-2403-44cf-bcd3-adcd19d5b448" (UID: "0b274ca5-2403-44cf-bcd3-adcd19d5b448"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:01:05.768245 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.768221    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0b274ca5-2403-44cf-bcd3-adcd19d5b448" (UID: "0b274ca5-2403-44cf-bcd3-adcd19d5b448"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:01:05.866643 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.866605    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-oauth-serving-cert\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:01:05.866643 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.866636    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-trusted-ca-bundle\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:01:05.866643 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.866649    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-serving-cert\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:01:05.866913 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.866662    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-config\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:01:05.866913 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.866675    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8m66\" (UniqueName: \"kubernetes.io/projected/0b274ca5-2403-44cf-bcd3-adcd19d5b448-kube-api-access-n8m66\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:01:05.866913 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.866688    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b274ca5-2403-44cf-bcd3-adcd19d5b448-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:01:05.866913 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:05.866699    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b274ca5-2403-44cf-bcd3-adcd19d5b448-console-oauth-config\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:01:06.276989 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:06.276956    2578 generic.go:356] "Generic (PLEG): container finished" podID="0b274ca5-2403-44cf-bcd3-adcd19d5b448" containerID="d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd" exitCode=0
Sep 15 19:01:06.277156 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:06.277001    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7dbf4dc-2pkvd"
Sep 15 19:01:06.277156 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:06.277027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7dbf4dc-2pkvd" event={"ID":"0b274ca5-2403-44cf-bcd3-adcd19d5b448","Type":"ContainerDied","Data":"d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd"}
Sep 15 19:01:06.277156 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:06.277076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7dbf4dc-2pkvd" event={"ID":"0b274ca5-2403-44cf-bcd3-adcd19d5b448","Type":"ContainerDied","Data":"051098f3adea743f83da1fdecbe9f657652e1b4c803a3d8c49f89d4fa4865a3d"}
Sep 15 19:01:06.277156 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:06.277095    2578 scope.go:117] "RemoveContainer" containerID="d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd"
Sep 15 19:01:06.285033 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:06.285020    2578 scope.go:117] "RemoveContainer" containerID="d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd"
Sep 15 19:01:06.285300 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:01:06.285277    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd\": container with ID starting with d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd not found: ID does not exist" containerID="d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd"
Sep 15 19:01:06.285349 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:06.285307    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd"} err="failed to get container status \"d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd\": rpc error: code = NotFound desc = could not find container \"d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd\": container with ID starting with d3a7d0b3ff0d0f48685081fc70f29167bd5dd80235acddfc58b67b9b536e78cd not found: ID does not exist"
Sep 15 19:01:06.292556 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:06.292534    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b7dbf4dc-2pkvd"]
Sep 15 19:01:06.300854 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:06.300832    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b7dbf4dc-2pkvd"]
Sep 15 19:01:08.098012 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:01:08.097980    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b274ca5-2403-44cf-bcd3-adcd19d5b448" path="/var/lib/kubelet/pods/0b274ca5-2403-44cf-bcd3-adcd19d5b448/volumes"
Sep 15 19:02:42.773484 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.773398    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:02:42.773979 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.773722    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b274ca5-2403-44cf-bcd3-adcd19d5b448" containerName="console"
Sep 15 19:02:42.773979 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.773737    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b274ca5-2403-44cf-bcd3-adcd19d5b448" containerName="console"
Sep 15 19:02:42.773979 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.773823    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b274ca5-2403-44cf-bcd3-adcd19d5b448" containerName="console"
Sep 15 19:02:42.776587 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.776571    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68"
Sep 15 19:02:42.778929 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.778905    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jsd5d\""
Sep 15 19:02:42.779044 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.779010    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 19:02:42.779510 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.779493    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 19:02:42.779577 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.779495    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 19:02:42.784286 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.784266    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:02:42.809845 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.809818    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:02:42.877736 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.877707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8nq\" (UniqueName: \"kubernetes.io/projected/f96f6a13-88c8-4a17-a26f-b665fc04e950-kube-api-access-vp8nq\") pod \"limitador-limitador-84994f7ffc-dfp68\" (UID: \"f96f6a13-88c8-4a17-a26f-b665fc04e950\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68"
Sep 15 19:02:42.877875 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.877744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f96f6a13-88c8-4a17-a26f-b665fc04e950-config-file\") pod \"limitador-limitador-84994f7ffc-dfp68\" (UID: \"f96f6a13-88c8-4a17-a26f-b665fc04e950\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68"
Sep 15 19:02:42.978869 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.978841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8nq\" (UniqueName: \"kubernetes.io/projected/f96f6a13-88c8-4a17-a26f-b665fc04e950-kube-api-access-vp8nq\") pod \"limitador-limitador-84994f7ffc-dfp68\" (UID: \"f96f6a13-88c8-4a17-a26f-b665fc04e950\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68"
Sep 15 19:02:42.978990 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.978885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f96f6a13-88c8-4a17-a26f-b665fc04e950-config-file\") pod \"limitador-limitador-84994f7ffc-dfp68\" (UID: \"f96f6a13-88c8-4a17-a26f-b665fc04e950\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68"
Sep 15 19:02:42.979531 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.979513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f96f6a13-88c8-4a17-a26f-b665fc04e950-config-file\") pod \"limitador-limitador-84994f7ffc-dfp68\" (UID: \"f96f6a13-88c8-4a17-a26f-b665fc04e950\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68"
Sep 15 19:02:42.986569 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:42.986551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8nq\" (UniqueName: \"kubernetes.io/projected/f96f6a13-88c8-4a17-a26f-b665fc04e950-kube-api-access-vp8nq\") pod \"limitador-limitador-84994f7ffc-dfp68\" (UID: \"f96f6a13-88c8-4a17-a26f-b665fc04e950\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68"
Sep 15 19:02:43.087449 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:43.087379    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68"
Sep 15 19:02:43.359046 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:43.358968    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:02:43.361750 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:02:43.361720    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96f6a13_88c8_4a17_a26f_b665fc04e950.slice/crio-a51538023a91cb0f6e4e67d3ca4c253f545a8c9c9276a1817732038173caaa09 WatchSource:0}: Error finding container a51538023a91cb0f6e4e67d3ca4c253f545a8c9c9276a1817732038173caaa09: Status 404 returned error can't find the container with id a51538023a91cb0f6e4e67d3ca4c253f545a8c9c9276a1817732038173caaa09
Sep 15 19:02:43.363690 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:43.363673    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:02:43.645411 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:43.645380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68" event={"ID":"f96f6a13-88c8-4a17-a26f-b665fc04e950","Type":"ContainerStarted","Data":"a51538023a91cb0f6e4e67d3ca4c253f545a8c9c9276a1817732038173caaa09"}
Sep 15 19:02:45.656232 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:45.656193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68" event={"ID":"f96f6a13-88c8-4a17-a26f-b665fc04e950","Type":"ContainerStarted","Data":"3103aea2edac7d3072579736ce9f775ea81495905663df627f3167bf45749358"}
Sep 15 19:02:45.656609 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:45.656382    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68"
Sep 15 19:02:45.671920 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:45.671851    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68" podStartSLOduration=1.495222448 podStartE2EDuration="3.671831032s" totalImagesPullingTime="2.176608584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:02:45.671018403 +0000 UTC m=+716.213113932" watchObservedRunningTime="2026-09-15 19:02:45.671831032 +0000 UTC m=+716.213926564"
Sep 15 19:02:56.661352 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:02:56.661321    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfp68"
Sep 15 19:03:59.895866 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:03:59.895828    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:04:33.476964 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:33.476876    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:04:49.980227 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:49.980195    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"]
Sep 15 19:04:49.983317 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:49.983295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:49.986548 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:49.986528    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 19:04:49.986673 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:49.986558    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 15 19:04:49.986673 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:49.986526    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 19:04:49.986673 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:49.986526    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bttfv\""
Sep 15 19:04:49.994629 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:49.994605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"]
Sep 15 19:04:50.012006 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.011980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhrt\" (UniqueName: \"kubernetes.io/projected/a6464af9-0104-4fc8-b55e-f733e59d654b-kube-api-access-gdhrt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.012104 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.012029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6464af9-0104-4fc8-b55e-f733e59d654b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.012104 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.012047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.012181 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.012127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.012181 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.012158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.012244 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.012179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.112740 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.112708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.112903 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.112751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhrt\" (UniqueName: \"kubernetes.io/projected/a6464af9-0104-4fc8-b55e-f733e59d654b-kube-api-access-gdhrt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.112903 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.112784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6464af9-0104-4fc8-b55e-f733e59d654b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.112903 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.112819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.112903 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.112843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.112903 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.112868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.113173 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.113148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.113173 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.113167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.113275 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.113204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.114936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.114913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6464af9-0104-4fc8-b55e-f733e59d654b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.115179 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.115162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6464af9-0104-4fc8-b55e-f733e59d654b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.121388 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.121363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhrt\" (UniqueName: \"kubernetes.io/projected/a6464af9-0104-4fc8-b55e-f733e59d654b-kube-api-access-gdhrt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-758wx\" (UID: \"a6464af9-0104-4fc8-b55e-f733e59d654b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.298066 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.297967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:04:50.567588 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.567564    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"]
Sep 15 19:04:50.569662 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:04:50.569635    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6464af9_0104_4fc8_b55e_f733e59d654b.slice/crio-7d352211551ab4d267a5025eed6d67d08a1c3c7c3d6ad960e0e9d507811579ee WatchSource:0}: Error finding container 7d352211551ab4d267a5025eed6d67d08a1c3c7c3d6ad960e0e9d507811579ee: Status 404 returned error can't find the container with id 7d352211551ab4d267a5025eed6d67d08a1c3c7c3d6ad960e0e9d507811579ee
Sep 15 19:04:50.671453 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:50.671421    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:04:51.145271 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:51.145228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx" event={"ID":"a6464af9-0104-4fc8-b55e-f733e59d654b","Type":"ContainerStarted","Data":"7d352211551ab4d267a5025eed6d67d08a1c3c7c3d6ad960e0e9d507811579ee"}
Sep 15 19:04:52.152146 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:52.152110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx" event={"ID":"a6464af9-0104-4fc8-b55e-f733e59d654b","Type":"ContainerStarted","Data":"e251a687ec592e1cca9f887e592a5130ab18d0be2256fcd0deb81b43ddfdcfd4"}
Sep 15 19:04:52.168332 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:04:52.168281    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx" podStartSLOduration=2.205896379 podStartE2EDuration="3.168264454s" totalImagesPullingTime="962.368075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:04:52.167830579 +0000 UTC m=+842.709926109" watchObservedRunningTime="2026-09-15 19:04:52.168264454 +0000 UTC m=+842.710359986"
Sep 15 19:05:00.298239 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:00.298206    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:05:00.298700 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:00.298260    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:05:00.310566 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:00.310543    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:05:01.199717 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:01.199685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-758wx"
Sep 15 19:05:04.770632 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:04.770595    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:05:13.676856 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:13.676791    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:05:26.080394 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:26.080357    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:05:32.578229 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.578189    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"]
Sep 15 19:05:32.581577 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.581554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.583825 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.583791    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 15 19:05:32.591329 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.591302    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"]
Sep 15 19:05:32.739410 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.739375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.739574 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.739424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.739574 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.739447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.739574 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.739481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.739574 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.739518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t5z6\" (UniqueName: \"kubernetes.io/projected/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-kube-api-access-7t5z6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.739710 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.739595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.840565 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.840493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.840565 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.840537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.840741 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.840579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.840741 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.840594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.840741 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.840609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.840741 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.840645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t5z6\" (UniqueName: \"kubernetes.io/projected/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-kube-api-access-7t5z6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.841036 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.841010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.841036 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.841025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.841174 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.841102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.842745 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.842726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.843157 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.843137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.848328 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.848298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t5z6\" (UniqueName: \"kubernetes.io/projected/7f5fb745-91ec-4963-bb0c-6ffb0ef1057c-kube-api-access-7t5z6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qvssg\" (UID: \"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:32.891970 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:32.891934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:33.162960 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:33.162928    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"]
Sep 15 19:05:33.166746 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:05:33.166720    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5fb745_91ec_4963_bb0c_6ffb0ef1057c.slice/crio-d06847da0593ae970dc8eb718b76baf1f2ab96079698f2c65124384a47c18098 WatchSource:0}: Error finding container d06847da0593ae970dc8eb718b76baf1f2ab96079698f2c65124384a47c18098: Status 404 returned error can't find the container with id d06847da0593ae970dc8eb718b76baf1f2ab96079698f2c65124384a47c18098
Sep 15 19:05:33.312711 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:33.312676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg" event={"ID":"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c","Type":"ContainerStarted","Data":"d06847da0593ae970dc8eb718b76baf1f2ab96079698f2c65124384a47c18098"}
Sep 15 19:05:34.324309 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:34.324271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg" event={"ID":"7f5fb745-91ec-4963-bb0c-6ffb0ef1057c","Type":"ContainerStarted","Data":"53cfb27de23e9efed358f718fe492f179f555e92a1429879a670586c3d4a3132"}
Sep 15 19:05:34.343340 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:34.343291    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg" podStartSLOduration=2.16213134 podStartE2EDuration="2.343276043s" totalImagesPullingTime="181.144703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:34.342335693 +0000 UTC m=+884.884431223" watchObservedRunningTime="2026-09-15 19:05:34.343276043 +0000 UTC m=+884.885371572"
Sep 15 19:05:34.385558 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:34.385530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:05:42.893037 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:42.892945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:42.893037 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:42.893023    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:42.905430 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:42.905408    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:43.376286 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:43.376259    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qvssg"
Sep 15 19:05:45.486856 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:05:45.486824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:07:44.925670 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:44.925591    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9011-df0a66"]
Sep 15 19:07:44.928818 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:44.928784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-df0a66"
Sep 15 19:07:44.931017 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:44.930995    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgkcn\""
Sep 15 19:07:44.935083 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:44.935058    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9011-df0a66"]
Sep 15 19:07:45.029562 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:45.029528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-547x5\" (UniqueName: \"kubernetes.io/projected/cc29c117-5899-466a-9bac-dd349200ef5e-kube-api-access-547x5\") pod \"test-curl-9011-df0a66\" (UID: \"cc29c117-5899-466a-9bac-dd349200ef5e\") " pod="opendatahub/test-curl-9011-df0a66"
Sep 15 19:07:45.130186 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:45.130156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-547x5\" (UniqueName: \"kubernetes.io/projected/cc29c117-5899-466a-9bac-dd349200ef5e-kube-api-access-547x5\") pod \"test-curl-9011-df0a66\" (UID: \"cc29c117-5899-466a-9bac-dd349200ef5e\") " pod="opendatahub/test-curl-9011-df0a66"
Sep 15 19:07:45.137394 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:45.137368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-547x5\" (UniqueName: \"kubernetes.io/projected/cc29c117-5899-466a-9bac-dd349200ef5e-kube-api-access-547x5\") pod \"test-curl-9011-df0a66\" (UID: \"cc29c117-5899-466a-9bac-dd349200ef5e\") " pod="opendatahub/test-curl-9011-df0a66"
Sep 15 19:07:45.238373 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:45.238297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-df0a66"
Sep 15 19:07:45.505583 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:45.505505    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9011-df0a66"]
Sep 15 19:07:45.508318 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:07:45.508289    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc29c117_5899_466a_9bac_dd349200ef5e.slice/crio-9c5d296e1c0d6e68f2965ced3053f51d2f1e618272d1382d6bbb4fa0eec3aed7 WatchSource:0}: Error finding container 9c5d296e1c0d6e68f2965ced3053f51d2f1e618272d1382d6bbb4fa0eec3aed7: Status 404 returned error can't find the container with id 9c5d296e1c0d6e68f2965ced3053f51d2f1e618272d1382d6bbb4fa0eec3aed7
Sep 15 19:07:45.510245 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:45.510227    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:07:45.859035 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:45.858946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9011-df0a66" event={"ID":"cc29c117-5899-466a-9bac-dd349200ef5e","Type":"ContainerStarted","Data":"9c5d296e1c0d6e68f2965ced3053f51d2f1e618272d1382d6bbb4fa0eec3aed7"}
Sep 15 19:07:48.878687 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:48.878644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9011-df0a66" event={"ID":"cc29c117-5899-466a-9bac-dd349200ef5e","Type":"ContainerStarted","Data":"022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec"}
Sep 15 19:07:48.894367 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:48.894319    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9011-df0a66" podStartSLOduration=2.575157999 podStartE2EDuration="4.894304756s" totalImagesPullingTime="2.319146757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:48.892320188 +0000 UTC m=+1019.434415716" watchObservedRunningTime="2026-09-15 19:07:48.894304756 +0000 UTC m=+1019.436400285"
Sep 15 19:07:48.905249 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:48.905213    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"]
Sep 15 19:07:48.908483 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:48.908464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:07:48.910553 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:48.910531    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9de44d\""
Sep 15 19:07:48.910646 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:48.910591    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9de44d-dockercfg-2lfn6\""
Sep 15 19:07:48.918767 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:48.918746    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"]
Sep 15 19:07:48.966654 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:48.966627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57p7q\" (UniqueName: \"kubernetes.io/projected/9ecfbe08-fd43-4fdf-9079-651189163fd3-kube-api-access-57p7q\") pod \"payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8\" (UID: \"9ecfbe08-fd43-4fdf-9079-651189163fd3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:07:48.966785 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:48.966681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ecfbe08-fd43-4fdf-9079-651189163fd3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8\" (UID: \"9ecfbe08-fd43-4fdf-9079-651189163fd3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:07:49.067244 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.067214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ecfbe08-fd43-4fdf-9079-651189163fd3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8\" (UID: \"9ecfbe08-fd43-4fdf-9079-651189163fd3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:07:49.067372 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.067286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57p7q\" (UniqueName: \"kubernetes.io/projected/9ecfbe08-fd43-4fdf-9079-651189163fd3-kube-api-access-57p7q\") pod \"payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8\" (UID: \"9ecfbe08-fd43-4fdf-9079-651189163fd3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:07:49.068031 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.068011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ecfbe08-fd43-4fdf-9079-651189163fd3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8\" (UID: \"9ecfbe08-fd43-4fdf-9079-651189163fd3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:07:49.075411 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.075381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57p7q\" (UniqueName: \"kubernetes.io/projected/9ecfbe08-fd43-4fdf-9079-651189163fd3-kube-api-access-57p7q\") pod \"payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8\" (UID: \"9ecfbe08-fd43-4fdf-9079-651189163fd3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:07:49.115927 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.115898    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"]
Sep 15 19:07:49.119547 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.119525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.121726 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.121702    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 19:07:49.121930 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.121874    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 19:07:49.121930 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.121874    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:07:49.122071 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.121941    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-f0ad35-serving-cert\""
Sep 15 19:07:49.122071 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.121957    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v55qf\""
Sep 15 19:07:49.132954 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.132897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"]
Sep 15 19:07:49.168205 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.168175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-metrics-certs\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.168324 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.168212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djn7g\" (UniqueName: \"kubernetes.io/projected/f1c72779-4ea8-4ef2-8344-a9c266150251-kube-api-access-djn7g\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.168324 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.168272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.168324 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.168316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c72779-4ea8-4ef2-8344-a9c266150251-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.218650 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.218623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:07:49.269025 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.268999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-metrics-certs\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.269115 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.269046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djn7g\" (UniqueName: \"kubernetes.io/projected/f1c72779-4ea8-4ef2-8344-a9c266150251-kube-api-access-djn7g\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.269115 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.269099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.269231 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.269136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c72779-4ea8-4ef2-8344-a9c266150251-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.269581 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:07:49.269556    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-serving-cert: secret "maas-api-e2e-worker-w3-f0ad35-serving-cert" not found
Sep 15 19:07:49.269658 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:07:49.269639    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-maas-api-tls podName:f1c72779-4ea8-4ef2-8344-a9c266150251 nodeName:}" failed. No retries permitted until 2026-09-15 19:07:49.769617106 +0000 UTC m=+1020.311712627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-maas-api-tls") pod "maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w" (UID: "f1c72779-4ea8-4ef2-8344-a9c266150251") : secret "maas-api-e2e-worker-w3-f0ad35-serving-cert" not found
Sep 15 19:07:49.270292 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.269953    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c72779-4ea8-4ef2-8344-a9c266150251-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.272627 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.272601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-metrics-certs\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.278904 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.278878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djn7g\" (UniqueName: \"kubernetes.io/projected/f1c72779-4ea8-4ef2-8344-a9c266150251-kube-api-access-djn7g\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.450472 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.450387    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9011-df0a66"]
Sep 15 19:07:49.454719 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.454677    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9011-df0a66"]
Sep 15 19:07:49.486449 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.486422    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"]
Sep 15 19:07:49.489858 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:07:49.489836    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecfbe08_fd43_4fdf_9079_651189163fd3.slice/crio-f5d8cf962f8246a746478f63f7d0b548eacb498c5568c362c8b0817310af559b WatchSource:0}: Error finding container f5d8cf962f8246a746478f63f7d0b548eacb498c5568c362c8b0817310af559b: Status 404 returned error can't find the container with id f5d8cf962f8246a746478f63f7d0b548eacb498c5568c362c8b0817310af559b
Sep 15 19:07:49.773129 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.773038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.775309 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.775283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:49.883488 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:49.883451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8" event={"ID":"9ecfbe08-fd43-4fdf-9079-651189163fd3","Type":"ContainerStarted","Data":"f5d8cf962f8246a746478f63f7d0b548eacb498c5568c362c8b0817310af559b"}
Sep 15 19:07:49.883882 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:07:49.883580    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9011-df0a66" container="test-curl-9011-df0a66"
Sep 15 19:07:50.034363 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.034293    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v55qf\""
Sep 15 19:07:50.043145 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.043121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:50.326418 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.326396    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"]
Sep 15 19:07:50.330893 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:07:50.330864    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c72779_4ea8_4ef2_8344_a9c266150251.slice/crio-06052d88de18e1ce89e24b29b8b2514858847eb5f004a390b9fd7e536eac005f WatchSource:0}: Error finding container 06052d88de18e1ce89e24b29b8b2514858847eb5f004a390b9fd7e536eac005f: Status 404 returned error can't find the container with id 06052d88de18e1ce89e24b29b8b2514858847eb5f004a390b9fd7e536eac005f
Sep 15 19:07:50.885014 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.884982    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"]
Sep 15 19:07:50.885364 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.885274    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc29c117-5899-466a-9bac-dd349200ef5e" containerName="test-curl-9011-df0a66"
Sep 15 19:07:50.885364 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.885285    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc29c117-5899-466a-9bac-dd349200ef5e" containerName="test-curl-9011-df0a66"
Sep 15 19:07:50.885364 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.885347    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc29c117-5899-466a-9bac-dd349200ef5e" containerName="test-curl-9011-df0a66"
Sep 15 19:07:50.888114 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.888094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:07:50.890178 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.890151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w" event={"ID":"f1c72779-4ea8-4ef2-8344-a9c266150251","Type":"ContainerStarted","Data":"06052d88de18e1ce89e24b29b8b2514858847eb5f004a390b9fd7e536eac005f"}
Sep 15 19:07:50.890292 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.890257    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-900c88\""
Sep 15 19:07:50.890337 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.890284    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9011-df0a66" podUID="cc29c117-5899-466a-9bac-dd349200ef5e" containerName="test-curl-9011-df0a66" containerID="cri-o://022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec" gracePeriod=2
Sep 15 19:07:50.897864 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.897838    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"]
Sep 15 19:07:50.945575 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.945551    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"]
Sep 15 19:07:50.949908 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.949892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:07:50.952022 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.952003    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-900c88-dockercfg-cc424\""
Sep 15 19:07:50.959653 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.959623    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"]
Sep 15 19:07:50.982904 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.982873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbj45\" (UniqueName: \"kubernetes.io/projected/f7954e7c-224c-4f41-aed2-d68b31550af8-kube-api-access-pbj45\") pod \"payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk\" (UID: \"f7954e7c-224c-4f41-aed2-d68b31550af8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:07:50.983050 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:50.982938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7954e7c-224c-4f41-aed2-d68b31550af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk\" (UID: \"f7954e7c-224c-4f41-aed2-d68b31550af8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:07:51.083484 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.083446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2\" (UID: \"dad6ea98-8625-427e-ad4e-5fb504ff8dd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:07:51.083664 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.083501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbj45\" (UniqueName: \"kubernetes.io/projected/f7954e7c-224c-4f41-aed2-d68b31550af8-kube-api-access-pbj45\") pod \"payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk\" (UID: \"f7954e7c-224c-4f41-aed2-d68b31550af8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:07:51.083664 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.083564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76ld\" (UniqueName: \"kubernetes.io/projected/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-kube-api-access-c76ld\") pod \"payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2\" (UID: \"dad6ea98-8625-427e-ad4e-5fb504ff8dd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:07:51.083664 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.083601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7954e7c-224c-4f41-aed2-d68b31550af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk\" (UID: \"f7954e7c-224c-4f41-aed2-d68b31550af8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:07:51.084351 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.084325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7954e7c-224c-4f41-aed2-d68b31550af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk\" (UID: \"f7954e7c-224c-4f41-aed2-d68b31550af8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:07:51.091276 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.091247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbj45\" (UniqueName: \"kubernetes.io/projected/f7954e7c-224c-4f41-aed2-d68b31550af8-kube-api-access-pbj45\") pod \"payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk\" (UID: \"f7954e7c-224c-4f41-aed2-d68b31550af8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:07:51.184541 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.184466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2\" (UID: \"dad6ea98-8625-427e-ad4e-5fb504ff8dd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:07:51.184692 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.184538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76ld\" (UniqueName: \"kubernetes.io/projected/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-kube-api-access-c76ld\") pod \"payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2\" (UID: \"dad6ea98-8625-427e-ad4e-5fb504ff8dd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:07:51.185312 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.185275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2\" (UID: \"dad6ea98-8625-427e-ad4e-5fb504ff8dd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:07:51.192682 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.192650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76ld\" (UniqueName: \"kubernetes.io/projected/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-kube-api-access-c76ld\") pod \"payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2\" (UID: \"dad6ea98-8625-427e-ad4e-5fb504ff8dd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:07:51.200911 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.200744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:07:51.260263 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.260225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:07:51.494234 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.494050    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"]
Sep 15 19:07:51.498377 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:07:51.498304    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7954e7c_224c_4f41_aed2_d68b31550af8.slice/crio-36cd649850326b6fcbb2438221a23f0434be81ec1912fb6c26c176ac661f659e WatchSource:0}: Error finding container 36cd649850326b6fcbb2438221a23f0434be81ec1912fb6c26c176ac661f659e: Status 404 returned error can't find the container with id 36cd649850326b6fcbb2438221a23f0434be81ec1912fb6c26c176ac661f659e
Sep 15 19:07:51.554926 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.554858    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"]
Sep 15 19:07:51.557786 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:07:51.557760    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad6ea98_8625_427e_ad4e_5fb504ff8dd0.slice/crio-7372d8f0d31658b360a06259184a1281fadea4dbf9bf422ff0e1037a7d2809e3 WatchSource:0}: Error finding container 7372d8f0d31658b360a06259184a1281fadea4dbf9bf422ff0e1037a7d2809e3: Status 404 returned error can't find the container with id 7372d8f0d31658b360a06259184a1281fadea4dbf9bf422ff0e1037a7d2809e3
Sep 15 19:07:51.899571 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.899532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk" event={"ID":"f7954e7c-224c-4f41-aed2-d68b31550af8","Type":"ContainerStarted","Data":"267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78"}
Sep 15 19:07:51.900060 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.899594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk" event={"ID":"f7954e7c-224c-4f41-aed2-d68b31550af8","Type":"ContainerStarted","Data":"36cd649850326b6fcbb2438221a23f0434be81ec1912fb6c26c176ac661f659e"}
Sep 15 19:07:51.900060 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.899754    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:07:51.902852 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.902457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8" event={"ID":"9ecfbe08-fd43-4fdf-9079-651189163fd3","Type":"ContainerStarted","Data":"11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52"}
Sep 15 19:07:51.903095 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.903038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:07:51.905107 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.905022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2" event={"ID":"dad6ea98-8625-427e-ad4e-5fb504ff8dd0","Type":"ContainerStarted","Data":"309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346"}
Sep 15 19:07:51.905107 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.905067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2" event={"ID":"dad6ea98-8625-427e-ad4e-5fb504ff8dd0","Type":"ContainerStarted","Data":"7372d8f0d31658b360a06259184a1281fadea4dbf9bf422ff0e1037a7d2809e3"}
Sep 15 19:07:51.905365 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.905324    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:07:51.915381 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.914896    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk" podStartSLOduration=1.914879269 podStartE2EDuration="1.914879269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:51.914637645 +0000 UTC m=+1022.456733178" watchObservedRunningTime="2026-09-15 19:07:51.914879269 +0000 UTC m=+1022.456974795"
Sep 15 19:07:51.945716 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.945304    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8" podStartSLOduration=2.533457982 podStartE2EDuration="3.94528588s" totalImagesPullingTime="1.411827898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:51.944950569 +0000 UTC m=+1022.487046104" watchObservedRunningTime="2026-09-15 19:07:51.94528588 +0000 UTC m=+1022.487381410"
Sep 15 19:07:51.974478 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:51.974045    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2" podStartSLOduration=1.974029914 podStartE2EDuration="1.974029914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:51.973956041 +0000 UTC m=+1022.516051574" watchObservedRunningTime="2026-09-15 19:07:51.974029914 +0000 UTC m=+1022.516125442"
Sep 15 19:07:52.986728 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:52.986701    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:07:53.111024 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.111003    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-df0a66"
Sep 15 19:07:53.202550 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.202474    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-547x5\" (UniqueName: \"kubernetes.io/projected/cc29c117-5899-466a-9bac-dd349200ef5e-kube-api-access-547x5\") pod \"cc29c117-5899-466a-9bac-dd349200ef5e\" (UID: \"cc29c117-5899-466a-9bac-dd349200ef5e\") "
Sep 15 19:07:53.204667 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.204633    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc29c117-5899-466a-9bac-dd349200ef5e-kube-api-access-547x5" (OuterVolumeSpecName: "kube-api-access-547x5") pod "cc29c117-5899-466a-9bac-dd349200ef5e" (UID: "cc29c117-5899-466a-9bac-dd349200ef5e"). InnerVolumeSpecName "kube-api-access-547x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:07:53.303820 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.303762    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-547x5\" (UniqueName: \"kubernetes.io/projected/cc29c117-5899-466a-9bac-dd349200ef5e-kube-api-access-547x5\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:07:53.915505 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.915469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w" event={"ID":"f1c72779-4ea8-4ef2-8344-a9c266150251","Type":"ContainerStarted","Data":"5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760"}
Sep 15 19:07:53.915697 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.915580    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:07:53.916751 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.916728    2578 generic.go:356] "Generic (PLEG): container finished" podID="cc29c117-5899-466a-9bac-dd349200ef5e" containerID="022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec" exitCode=137
Sep 15 19:07:53.916863 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.916773    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9011-df0a66"
Sep 15 19:07:53.916863 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.916838    2578 scope.go:117] "RemoveContainer" containerID="022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec"
Sep 15 19:07:53.925031 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.925011    2578 scope.go:117] "RemoveContainer" containerID="022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec"
Sep 15 19:07:53.925292 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:07:53.925262    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec\": container with ID starting with 022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec not found: ID does not exist" containerID="022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec"
Sep 15 19:07:53.925409 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.925300    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec"} err="failed to get container status \"022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec\": rpc error: code = NotFound desc = could not find container \"022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec\": container with ID starting with 022a48c872092f10f4c1525f2969dd099448ede7c4a265b870a092607de5eeec not found: ID does not exist"
Sep 15 19:07:53.932281 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:07:53.932254    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9011-df0a66\" is forbidden: User \"system:node:ip-10-0-135-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-221.ec2.internal' and this object" podUID="cc29c117-5899-466a-9bac-dd349200ef5e" pod="opendatahub/test-curl-9011-df0a66"
Sep 15 19:07:53.932608 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:53.932571    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w" podStartSLOduration=2.233501898 podStartE2EDuration="4.932558728s" totalImagesPullingTime="2.69905683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:53.930461097 +0000 UTC m=+1024.472556626" watchObservedRunningTime="2026-09-15 19:07:53.932558728 +0000 UTC m=+1024.474654256"
Sep 15 19:07:54.097750 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:54.097716    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc29c117-5899-466a-9bac-dd349200ef5e" path="/var/lib/kubelet/pods/cc29c117-5899-466a-9bac-dd349200ef5e/volumes"
Sep 15 19:07:57.832241 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:57.832210    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"]
Sep 15 19:07:57.837035 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:57.837016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:07:57.839190 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:57.839172    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a7a10c67\""
Sep 15 19:07:57.839340 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:57.839267    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a7a10c67-dockercfg-z9h9d\""
Sep 15 19:07:57.844454 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:57.844405    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"]
Sep 15 19:07:57.941355 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:57.941326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jprh\" (UniqueName: \"kubernetes.io/projected/42e83377-bf12-4197-ba88-006cb35ce52c-kube-api-access-2jprh\") pod \"payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg\" (UID: \"42e83377-bf12-4197-ba88-006cb35ce52c\") " pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:07:57.941509 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:57.941361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e83377-bf12-4197-ba88-006cb35ce52c-plugins-config-volume\") pod \"payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg\" (UID: \"42e83377-bf12-4197-ba88-006cb35ce52c\") " pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:07:58.041968 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:58.041937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jprh\" (UniqueName: \"kubernetes.io/projected/42e83377-bf12-4197-ba88-006cb35ce52c-kube-api-access-2jprh\") pod \"payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg\" (UID: \"42e83377-bf12-4197-ba88-006cb35ce52c\") " pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:07:58.041968 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:58.041969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e83377-bf12-4197-ba88-006cb35ce52c-plugins-config-volume\") pod \"payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg\" (UID: \"42e83377-bf12-4197-ba88-006cb35ce52c\") " pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:07:58.042681 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:58.042662    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e83377-bf12-4197-ba88-006cb35ce52c-plugins-config-volume\") pod \"payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg\" (UID: \"42e83377-bf12-4197-ba88-006cb35ce52c\") " pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:07:58.049639 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:58.049609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jprh\" (UniqueName: \"kubernetes.io/projected/42e83377-bf12-4197-ba88-006cb35ce52c-kube-api-access-2jprh\") pod \"payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg\" (UID: \"42e83377-bf12-4197-ba88-006cb35ce52c\") " pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:07:58.148287 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:58.148266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:07:58.418334 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:58.418269    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"]
Sep 15 19:07:58.422070 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:07:58.422040    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e83377_bf12_4197_ba88_006cb35ce52c.slice/crio-c157068edc462bbb6e33c15c496d1113cdb15c2043f6050d3bc7a3d41ba77450 WatchSource:0}: Error finding container c157068edc462bbb6e33c15c496d1113cdb15c2043f6050d3bc7a3d41ba77450: Status 404 returned error can't find the container with id c157068edc462bbb6e33c15c496d1113cdb15c2043f6050d3bc7a3d41ba77450
Sep 15 19:07:58.941159 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:58.941125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg" event={"ID":"42e83377-bf12-4197-ba88-006cb35ce52c","Type":"ContainerStarted","Data":"e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31"}
Sep 15 19:07:58.941541 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:58.941173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg" event={"ID":"42e83377-bf12-4197-ba88-006cb35ce52c","Type":"ContainerStarted","Data":"c157068edc462bbb6e33c15c496d1113cdb15c2043f6050d3bc7a3d41ba77450"}
Sep 15 19:07:58.941541 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:58.941237    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:07:58.957859 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:58.957790    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg" podStartSLOduration=1.957776946 podStartE2EDuration="1.957776946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:58.957320806 +0000 UTC m=+1029.499416336" watchObservedRunningTime="2026-09-15 19:07:58.957776946 +0000 UTC m=+1029.499872474"
Sep 15 19:07:59.927560 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:07:59.927523    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:08:02.911269 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:02.911232    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:08:02.911707 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:02.911448    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:08:03.918922 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:03.918889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:08:09.948658 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:09.948627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:08:16.473951 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:16.473899    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:08:19.170120 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.170078    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"]
Sep 15 19:08:19.173291 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.173275    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:08:19.176411 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.176388    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3daa58\""
Sep 15 19:08:19.176534 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.176473    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3daa58-dockercfg-vdgns\""
Sep 15 19:08:19.188164 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.188140    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"]
Sep 15 19:08:19.190135 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.190113    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"]
Sep 15 19:08:19.193222 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.193205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:08:19.203533 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.203507    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"]
Sep 15 19:08:19.314630 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.314589    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6pp\" (UniqueName: \"kubernetes.io/projected/4759ff24-1def-4fdd-b519-113c28254f04-kube-api-access-9d6pp\") pod \"payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct\" (UID: \"4759ff24-1def-4fdd-b519-113c28254f04\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:08:19.314813 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.314641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7fx\" (UniqueName: \"kubernetes.io/projected/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-kube-api-access-jg7fx\") pod \"payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj\" (UID: \"03ff452a-fd2d-4f06-873e-c1a98c94cb3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:08:19.314813 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.314753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4759ff24-1def-4fdd-b519-113c28254f04-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct\" (UID: \"4759ff24-1def-4fdd-b519-113c28254f04\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:08:19.314935 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.314813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj\" (UID: \"03ff452a-fd2d-4f06-873e-c1a98c94cb3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:08:19.415964 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.415916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4759ff24-1def-4fdd-b519-113c28254f04-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct\" (UID: \"4759ff24-1def-4fdd-b519-113c28254f04\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:08:19.415964 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.415970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj\" (UID: \"03ff452a-fd2d-4f06-873e-c1a98c94cb3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:08:19.416202 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.416003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6pp\" (UniqueName: \"kubernetes.io/projected/4759ff24-1def-4fdd-b519-113c28254f04-kube-api-access-9d6pp\") pod \"payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct\" (UID: \"4759ff24-1def-4fdd-b519-113c28254f04\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:08:19.416202 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.416025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7fx\" (UniqueName: \"kubernetes.io/projected/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-kube-api-access-jg7fx\") pod \"payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj\" (UID: \"03ff452a-fd2d-4f06-873e-c1a98c94cb3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:08:19.416704 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.416678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4759ff24-1def-4fdd-b519-113c28254f04-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct\" (UID: \"4759ff24-1def-4fdd-b519-113c28254f04\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:08:19.416837 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.416754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj\" (UID: \"03ff452a-fd2d-4f06-873e-c1a98c94cb3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:08:19.424605 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.424554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7fx\" (UniqueName: \"kubernetes.io/projected/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-kube-api-access-jg7fx\") pod \"payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj\" (UID: \"03ff452a-fd2d-4f06-873e-c1a98c94cb3b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:08:19.424605 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.424573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6pp\" (UniqueName: \"kubernetes.io/projected/4759ff24-1def-4fdd-b519-113c28254f04-kube-api-access-9d6pp\") pod \"payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct\" (UID: \"4759ff24-1def-4fdd-b519-113c28254f04\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:08:19.483090 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.483063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:08:19.503138 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.503117    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:08:19.755485 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.755417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"]
Sep 15 19:08:19.758691 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:08:19.758664    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ff452a_fd2d_4f06_873e_c1a98c94cb3b.slice/crio-507509f4edd7d64a96ca4e5c8b60f26c78ce2f36ca70634f345bc3c60c7825ef WatchSource:0}: Error finding container 507509f4edd7d64a96ca4e5c8b60f26c78ce2f36ca70634f345bc3c60c7825ef: Status 404 returned error can't find the container with id 507509f4edd7d64a96ca4e5c8b60f26c78ce2f36ca70634f345bc3c60c7825ef
Sep 15 19:08:19.792247 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:19.792198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"]
Sep 15 19:08:19.799833 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:08:19.799788    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4759ff24_1def_4fdd_b519_113c28254f04.slice/crio-5c027ff4755cfffb6a9d4d46a4584d6923e49c21da31fa0d49aa3235bccaad08 WatchSource:0}: Error finding container 5c027ff4755cfffb6a9d4d46a4584d6923e49c21da31fa0d49aa3235bccaad08: Status 404 returned error can't find the container with id 5c027ff4755cfffb6a9d4d46a4584d6923e49c21da31fa0d49aa3235bccaad08
Sep 15 19:08:20.039441 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:20.039355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj" event={"ID":"03ff452a-fd2d-4f06-873e-c1a98c94cb3b","Type":"ContainerStarted","Data":"6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff"}
Sep 15 19:08:20.039441 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:20.039420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj" event={"ID":"03ff452a-fd2d-4f06-873e-c1a98c94cb3b","Type":"ContainerStarted","Data":"507509f4edd7d64a96ca4e5c8b60f26c78ce2f36ca70634f345bc3c60c7825ef"}
Sep 15 19:08:20.039657 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:20.039451    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:08:20.041090 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:20.041063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct" event={"ID":"4759ff24-1def-4fdd-b519-113c28254f04","Type":"ContainerStarted","Data":"a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90"}
Sep 15 19:08:20.041213 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:20.041107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct" event={"ID":"4759ff24-1def-4fdd-b519-113c28254f04","Type":"ContainerStarted","Data":"5c027ff4755cfffb6a9d4d46a4584d6923e49c21da31fa0d49aa3235bccaad08"}
Sep 15 19:08:20.041269 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:20.041218    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:08:20.056440 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:20.056390    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj" podStartSLOduration=1.056376927 podStartE2EDuration="1.056376927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:20.054270323 +0000 UTC m=+1050.596365852" watchObservedRunningTime="2026-09-15 19:08:20.056376927 +0000 UTC m=+1050.598472456"
Sep 15 19:08:20.070854 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:20.070811    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct" podStartSLOduration=1.07078227 podStartE2EDuration="1.07078227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:20.069509747 +0000 UTC m=+1050.611605275" watchObservedRunningTime="2026-09-15 19:08:20.07078227 +0000 UTC m=+1050.612877798"
Sep 15 19:08:25.528915 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:25.528870    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"]
Sep 15 19:08:25.529509 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:25.529100    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg" podUID="42e83377-bf12-4197-ba88-006cb35ce52c" containerName="payload-processing" containerID="cri-o://e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31" gracePeriod=30
Sep 15 19:08:25.766960 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:25.766934    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:08:25.856835 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:25.856737    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e83377-bf12-4197-ba88-006cb35ce52c-plugins-config-volume\") pod \"42e83377-bf12-4197-ba88-006cb35ce52c\" (UID: \"42e83377-bf12-4197-ba88-006cb35ce52c\") "
Sep 15 19:08:25.856835 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:25.856824    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jprh\" (UniqueName: \"kubernetes.io/projected/42e83377-bf12-4197-ba88-006cb35ce52c-kube-api-access-2jprh\") pod \"42e83377-bf12-4197-ba88-006cb35ce52c\" (UID: \"42e83377-bf12-4197-ba88-006cb35ce52c\") "
Sep 15 19:08:25.857201 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:25.857169    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e83377-bf12-4197-ba88-006cb35ce52c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42e83377-bf12-4197-ba88-006cb35ce52c" (UID: "42e83377-bf12-4197-ba88-006cb35ce52c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:08:25.858949 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:25.858915    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e83377-bf12-4197-ba88-006cb35ce52c-kube-api-access-2jprh" (OuterVolumeSpecName: "kube-api-access-2jprh") pod "42e83377-bf12-4197-ba88-006cb35ce52c" (UID: "42e83377-bf12-4197-ba88-006cb35ce52c"). InnerVolumeSpecName "kube-api-access-2jprh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:08:25.957837 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:25.957805    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e83377-bf12-4197-ba88-006cb35ce52c-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:08:25.957837 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:25.957832    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jprh\" (UniqueName: \"kubernetes.io/projected/42e83377-bf12-4197-ba88-006cb35ce52c-kube-api-access-2jprh\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:08:26.072088 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:26.072049    2578 generic.go:356] "Generic (PLEG): container finished" podID="42e83377-bf12-4197-ba88-006cb35ce52c" containerID="e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31" exitCode=0
Sep 15 19:08:26.072258 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:26.072115    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"
Sep 15 19:08:26.072258 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:26.072125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg" event={"ID":"42e83377-bf12-4197-ba88-006cb35ce52c","Type":"ContainerDied","Data":"e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31"}
Sep 15 19:08:26.072258 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:26.072210    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg" event={"ID":"42e83377-bf12-4197-ba88-006cb35ce52c","Type":"ContainerDied","Data":"c157068edc462bbb6e33c15c496d1113cdb15c2043f6050d3bc7a3d41ba77450"}
Sep 15 19:08:26.072258 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:26.072240    2578 scope.go:117] "RemoveContainer" containerID="e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31"
Sep 15 19:08:26.080844 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:26.080825    2578 scope.go:117] "RemoveContainer" containerID="e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31"
Sep 15 19:08:26.081105 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:08:26.081083    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31\": container with ID starting with e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31 not found: ID does not exist" containerID="e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31"
Sep 15 19:08:26.081153 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:26.081113    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31"} err="failed to get container status \"e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31\": rpc error: code = NotFound desc = could not find container \"e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31\": container with ID starting with e5eb08d41a42edd0e6c1910a1fffe9502dab6b650b2034cdd19577c5811e0a31 not found: ID does not exist"
Sep 15 19:08:26.099847 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:26.099821    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"]
Sep 15 19:08:26.105626 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:26.105603    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a7a10c67-5cc7bc6695-mcjrg"]
Sep 15 19:08:28.098421 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:28.098390    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e83377-bf12-4197-ba88-006cb35ce52c" path="/var/lib/kubelet/pods/42e83377-bf12-4197-ba88-006cb35ce52c/volumes"
Sep 15 19:08:30.200075 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.200042    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"]
Sep 15 19:08:30.200447 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.200372    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e83377-bf12-4197-ba88-006cb35ce52c" containerName="payload-processing"
Sep 15 19:08:30.200447 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.200384    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e83377-bf12-4197-ba88-006cb35ce52c" containerName="payload-processing"
Sep 15 19:08:30.200518 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.200457    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e83377-bf12-4197-ba88-006cb35ce52c" containerName="payload-processing"
Sep 15 19:08:30.204688 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.204658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.206792 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.206771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8b287f5b-serving-cert\""
Sep 15 19:08:30.215543 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.215518    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"]
Sep 15 19:08:30.288751 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.288716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pwjc\" (UniqueName: \"kubernetes.io/projected/99b99287-c3ab-4fea-ab24-5b4518354bc3-kube-api-access-5pwjc\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.288751 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.288750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-maas-api-tls\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.289001 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.288768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-metrics-certs\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.289001 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.288914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99b99287-c3ab-4fea-ab24-5b4518354bc3-openshift-service-ca\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.389354 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.389320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-maas-api-tls\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.389354 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.389353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-metrics-certs\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.389550 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.389396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99b99287-c3ab-4fea-ab24-5b4518354bc3-openshift-service-ca\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.389550 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.389434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pwjc\" (UniqueName: \"kubernetes.io/projected/99b99287-c3ab-4fea-ab24-5b4518354bc3-kube-api-access-5pwjc\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.389550 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:08:30.389489    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-serving-cert: secret "maas-api-e2e-migrate-8b287f5b-serving-cert" not found
Sep 15 19:08:30.389666 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:08:30.389595    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-maas-api-tls podName:99b99287-c3ab-4fea-ab24-5b4518354bc3 nodeName:}" failed. No retries permitted until 2026-09-15 19:08:30.889574976 +0000 UTC m=+1061.431670484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-maas-api-tls") pod "maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln" (UID: "99b99287-c3ab-4fea-ab24-5b4518354bc3") : secret "maas-api-e2e-migrate-8b287f5b-serving-cert" not found
Sep 15 19:08:30.390033 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.390007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99b99287-c3ab-4fea-ab24-5b4518354bc3-openshift-service-ca\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.391903 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.391884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-metrics-certs\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.396923 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.396902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pwjc\" (UniqueName: \"kubernetes.io/projected/99b99287-c3ab-4fea-ab24-5b4518354bc3-kube-api-access-5pwjc\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.817582 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.817547    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"]
Sep 15 19:08:30.821061 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.821038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:08:30.823317 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.823289    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8b287f5b\""
Sep 15 19:08:30.823417 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.823385    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8b287f5b-dockercfg-tkxr6\""
Sep 15 19:08:30.831334 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.831314    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"]
Sep 15 19:08:30.893106 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.893079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-maas-api-tls\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.893243 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.893126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38394cf-f0e9-4864-b07d-1e14fa12c66d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb\" (UID: \"c38394cf-f0e9-4864-b07d-1e14fa12c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:08:30.893287 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.893256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxw6\" (UniqueName: \"kubernetes.io/projected/c38394cf-f0e9-4864-b07d-1e14fa12c66d-kube-api-access-sdxw6\") pod \"payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb\" (UID: \"c38394cf-f0e9-4864-b07d-1e14fa12c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:08:30.895580 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.895550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-maas-api-tls\") pod \"maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:30.994474 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.994442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38394cf-f0e9-4864-b07d-1e14fa12c66d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb\" (UID: \"c38394cf-f0e9-4864-b07d-1e14fa12c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:08:30.994626 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.994483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxw6\" (UniqueName: \"kubernetes.io/projected/c38394cf-f0e9-4864-b07d-1e14fa12c66d-kube-api-access-sdxw6\") pod \"payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb\" (UID: \"c38394cf-f0e9-4864-b07d-1e14fa12c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:08:30.995179 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:30.995161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38394cf-f0e9-4864-b07d-1e14fa12c66d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb\" (UID: \"c38394cf-f0e9-4864-b07d-1e14fa12c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:08:31.001852 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:31.001830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxw6\" (UniqueName: \"kubernetes.io/projected/c38394cf-f0e9-4864-b07d-1e14fa12c66d-kube-api-access-sdxw6\") pod \"payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb\" (UID: \"c38394cf-f0e9-4864-b07d-1e14fa12c66d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:08:31.047454 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:31.047434    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:08:31.047561 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:31.047501    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:08:31.118204 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:31.118129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:31.132282 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:31.132255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:08:31.407012 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:31.404363    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"]
Sep 15 19:08:31.412371 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:08:31.412345    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b99287_c3ab_4fea_ab24_5b4518354bc3.slice/crio-c2738e3220095fb1f4ed6f9c8d279ace78706fc54d2946cd6c8ee9be9c2ee874 WatchSource:0}: Error finding container c2738e3220095fb1f4ed6f9c8d279ace78706fc54d2946cd6c8ee9be9c2ee874: Status 404 returned error can't find the container with id c2738e3220095fb1f4ed6f9c8d279ace78706fc54d2946cd6c8ee9be9c2ee874
Sep 15 19:08:31.425214 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:31.425168    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"]
Sep 15 19:08:31.427054 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:08:31.427031    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38394cf_f0e9_4864_b07d_1e14fa12c66d.slice/crio-4d16d6aab0ad8f29b77e374227beaf3a2599a4514f10248537498238b17cf0bc WatchSource:0}: Error finding container 4d16d6aab0ad8f29b77e374227beaf3a2599a4514f10248537498238b17cf0bc: Status 404 returned error can't find the container with id 4d16d6aab0ad8f29b77e374227beaf3a2599a4514f10248537498238b17cf0bc
Sep 15 19:08:32.101824 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:32.101730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb" event={"ID":"c38394cf-f0e9-4864-b07d-1e14fa12c66d","Type":"ContainerStarted","Data":"f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4"}
Sep 15 19:08:32.101824 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:32.101810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb" event={"ID":"c38394cf-f0e9-4864-b07d-1e14fa12c66d","Type":"ContainerStarted","Data":"4d16d6aab0ad8f29b77e374227beaf3a2599a4514f10248537498238b17cf0bc"}
Sep 15 19:08:32.102068 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:32.101845    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:08:32.103319 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:32.103293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln" event={"ID":"99b99287-c3ab-4fea-ab24-5b4518354bc3","Type":"ContainerStarted","Data":"fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba"}
Sep 15 19:08:32.103430 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:32.103331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln" event={"ID":"99b99287-c3ab-4fea-ab24-5b4518354bc3","Type":"ContainerStarted","Data":"c2738e3220095fb1f4ed6f9c8d279ace78706fc54d2946cd6c8ee9be9c2ee874"}
Sep 15 19:08:32.103430 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:32.103416    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:32.116372 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:32.116326    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb" podStartSLOduration=2.116314059 podStartE2EDuration="2.116314059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:32.114996709 +0000 UTC m=+1062.657092237" watchObservedRunningTime="2026-09-15 19:08:32.116314059 +0000 UTC m=+1062.658409585"
Sep 15 19:08:32.131147 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:32.131106    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln" podStartSLOduration=1.718785655 podStartE2EDuration="2.131093975s" totalImagesPullingTime="412.30832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:32.129613071 +0000 UTC m=+1062.671708599" watchObservedRunningTime="2026-09-15 19:08:32.131093975 +0000 UTC m=+1062.673189533"
Sep 15 19:08:38.113921 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:38.113892    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:08:41.076002 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:41.075960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:08:43.110264 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:08:43.110181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:09:01.641358 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:01.641314    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"]
Sep 15 19:09:01.642109 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:01.641566    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln" podUID="99b99287-c3ab-4fea-ab24-5b4518354bc3" containerName="maas-api" containerID="cri-o://fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba" gracePeriod=30
Sep 15 19:09:01.699484 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:01.699452    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"]
Sep 15 19:09:01.699700 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:01.699673    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb" podUID="c38394cf-f0e9-4864-b07d-1e14fa12c66d" containerName="payload-pre-processing" containerID="cri-o://f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4" gracePeriod=30
Sep 15 19:09:01.958726 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:01.958638    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:09:01.990842 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:01.990792    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:09:02.044723 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.044689    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-maas-api-tls\") pod \"99b99287-c3ab-4fea-ab24-5b4518354bc3\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") "
Sep 15 19:09:02.044723 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.044728    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99b99287-c3ab-4fea-ab24-5b4518354bc3-openshift-service-ca\") pod \"99b99287-c3ab-4fea-ab24-5b4518354bc3\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") "
Sep 15 19:09:02.045010 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.044756    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38394cf-f0e9-4864-b07d-1e14fa12c66d-plugins-config-volume\") pod \"c38394cf-f0e9-4864-b07d-1e14fa12c66d\" (UID: \"c38394cf-f0e9-4864-b07d-1e14fa12c66d\") "
Sep 15 19:09:02.045010 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.044894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdxw6\" (UniqueName: \"kubernetes.io/projected/c38394cf-f0e9-4864-b07d-1e14fa12c66d-kube-api-access-sdxw6\") pod \"c38394cf-f0e9-4864-b07d-1e14fa12c66d\" (UID: \"c38394cf-f0e9-4864-b07d-1e14fa12c66d\") "
Sep 15 19:09:02.045010 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.044954    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pwjc\" (UniqueName: \"kubernetes.io/projected/99b99287-c3ab-4fea-ab24-5b4518354bc3-kube-api-access-5pwjc\") pod \"99b99287-c3ab-4fea-ab24-5b4518354bc3\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") "
Sep 15 19:09:02.045163 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.045013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-metrics-certs\") pod \"99b99287-c3ab-4fea-ab24-5b4518354bc3\" (UID: \"99b99287-c3ab-4fea-ab24-5b4518354bc3\") "
Sep 15 19:09:02.045163 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.045104    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b99287-c3ab-4fea-ab24-5b4518354bc3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99b99287-c3ab-4fea-ab24-5b4518354bc3" (UID: "99b99287-c3ab-4fea-ab24-5b4518354bc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:02.045270 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.045172    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38394cf-f0e9-4864-b07d-1e14fa12c66d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c38394cf-f0e9-4864-b07d-1e14fa12c66d" (UID: "c38394cf-f0e9-4864-b07d-1e14fa12c66d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:02.045349 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.045332    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99b99287-c3ab-4fea-ab24-5b4518354bc3-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:02.045409 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.045351    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38394cf-f0e9-4864-b07d-1e14fa12c66d-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:02.047110 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.047082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b99287-c3ab-4fea-ab24-5b4518354bc3-kube-api-access-5pwjc" (OuterVolumeSpecName: "kube-api-access-5pwjc") pod "99b99287-c3ab-4fea-ab24-5b4518354bc3" (UID: "99b99287-c3ab-4fea-ab24-5b4518354bc3"). InnerVolumeSpecName "kube-api-access-5pwjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:02.047215 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.047165    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99b99287-c3ab-4fea-ab24-5b4518354bc3" (UID: "99b99287-c3ab-4fea-ab24-5b4518354bc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:02.047328 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.047309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "99b99287-c3ab-4fea-ab24-5b4518354bc3" (UID: "99b99287-c3ab-4fea-ab24-5b4518354bc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:02.047406 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.047391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38394cf-f0e9-4864-b07d-1e14fa12c66d-kube-api-access-sdxw6" (OuterVolumeSpecName: "kube-api-access-sdxw6") pod "c38394cf-f0e9-4864-b07d-1e14fa12c66d" (UID: "c38394cf-f0e9-4864-b07d-1e14fa12c66d"). InnerVolumeSpecName "kube-api-access-sdxw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:02.145890 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.145854    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdxw6\" (UniqueName: \"kubernetes.io/projected/c38394cf-f0e9-4864-b07d-1e14fa12c66d-kube-api-access-sdxw6\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:02.145890 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.145883    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pwjc\" (UniqueName: \"kubernetes.io/projected/99b99287-c3ab-4fea-ab24-5b4518354bc3-kube-api-access-5pwjc\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:02.145890 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.145893    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:02.146141 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.145904    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99b99287-c3ab-4fea-ab24-5b4518354bc3-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:02.240009 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.239929    2578 generic.go:356] "Generic (PLEG): container finished" podID="c38394cf-f0e9-4864-b07d-1e14fa12c66d" containerID="f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4" exitCode=0
Sep 15 19:09:02.240009 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.239974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"
Sep 15 19:09:02.240009 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.240006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb" event={"ID":"c38394cf-f0e9-4864-b07d-1e14fa12c66d","Type":"ContainerDied","Data":"f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4"}
Sep 15 19:09:02.240275 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.240058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb" event={"ID":"c38394cf-f0e9-4864-b07d-1e14fa12c66d","Type":"ContainerDied","Data":"4d16d6aab0ad8f29b77e374227beaf3a2599a4514f10248537498238b17cf0bc"}
Sep 15 19:09:02.240275 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.240091    2578 scope.go:117] "RemoveContainer" containerID="f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4"
Sep 15 19:09:02.241617 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.241588    2578 generic.go:356] "Generic (PLEG): container finished" podID="99b99287-c3ab-4fea-ab24-5b4518354bc3" containerID="fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba" exitCode=0
Sep 15 19:09:02.241745 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.241633    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"
Sep 15 19:09:02.241745 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.241673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln" event={"ID":"99b99287-c3ab-4fea-ab24-5b4518354bc3","Type":"ContainerDied","Data":"fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba"}
Sep 15 19:09:02.241745 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.241730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln" event={"ID":"99b99287-c3ab-4fea-ab24-5b4518354bc3","Type":"ContainerDied","Data":"c2738e3220095fb1f4ed6f9c8d279ace78706fc54d2946cd6c8ee9be9c2ee874"}
Sep 15 19:09:02.248698 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.248679    2578 scope.go:117] "RemoveContainer" containerID="f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4"
Sep 15 19:09:02.249005 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:09:02.248982    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4\": container with ID starting with f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4 not found: ID does not exist" containerID="f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4"
Sep 15 19:09:02.249101 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.249009    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4"} err="failed to get container status \"f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4\": rpc error: code = NotFound desc = could not find container \"f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4\": container with ID starting with f6798b12ec3094f35d1e8d874557bb2179a9a610b1544440895c40ce9ee3f0a4 not found: ID does not exist"
Sep 15 19:09:02.249101 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.249025    2578 scope.go:117] "RemoveContainer" containerID="fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba"
Sep 15 19:09:02.257736 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.257715    2578 scope.go:117] "RemoveContainer" containerID="fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba"
Sep 15 19:09:02.257874 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.257746    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"]
Sep 15 19:09:02.258656 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:09:02.258634    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba\": container with ID starting with fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba not found: ID does not exist" containerID="fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba"
Sep 15 19:09:02.258742 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.258660    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba"} err="failed to get container status \"fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba\": rpc error: code = NotFound desc = could not find container \"fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba\": container with ID starting with fde3f6b6a50ce8f69568fbcb08cdd1b89c60a6f6081b124a8cf7f0d1a642b6ba not found: ID does not exist"
Sep 15 19:09:02.260357 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.260338    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8b287f5b-78f8d5bbd5-9fvln"]
Sep 15 19:09:02.271586 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.271556    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"]
Sep 15 19:09:02.277945 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:02.277924    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8b287f5b-5bbb9fcbbb-bfjjb"]
Sep 15 19:09:04.098991 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:04.098958    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b99287-c3ab-4fea-ab24-5b4518354bc3" path="/var/lib/kubelet/pods/99b99287-c3ab-4fea-ab24-5b4518354bc3/volumes"
Sep 15 19:09:04.099377 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:04.099313    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c38394cf-f0e9-4864-b07d-1e14fa12c66d" path="/var/lib/kubelet/pods/c38394cf-f0e9-4864-b07d-1e14fa12c66d/volumes"
Sep 15 19:09:07.574251 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:07.574217    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:09:08.182291 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.182255    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"]
Sep 15 19:09:08.182652 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.182634    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c38394cf-f0e9-4864-b07d-1e14fa12c66d" containerName="payload-pre-processing"
Sep 15 19:09:08.182743 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.182653    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c38394cf-f0e9-4864-b07d-1e14fa12c66d" containerName="payload-pre-processing"
Sep 15 19:09:08.182743 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.182685    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b99287-c3ab-4fea-ab24-5b4518354bc3" containerName="maas-api"
Sep 15 19:09:08.182743 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.182694    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b99287-c3ab-4fea-ab24-5b4518354bc3" containerName="maas-api"
Sep 15 19:09:08.183065 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.182776    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b99287-c3ab-4fea-ab24-5b4518354bc3" containerName="maas-api"
Sep 15 19:09:08.183065 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.182788    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c38394cf-f0e9-4864-b07d-1e14fa12c66d" containerName="payload-pre-processing"
Sep 15 19:09:08.187282 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.187260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.189616 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.189594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-140d8435-serving-cert\""
Sep 15 19:09:08.196591 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.196565    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"]
Sep 15 19:09:08.296860 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.296825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56edc13f-f855-4642-9f9f-2d1e96792a87-openshift-service-ca\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.297007 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.296889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-maas-api-tls\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.297007 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.296923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-metrics-certs\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.297007 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.296985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7wz\" (UniqueName: \"kubernetes.io/projected/56edc13f-f855-4642-9f9f-2d1e96792a87-kube-api-access-rp7wz\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.397555 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.397525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56edc13f-f855-4642-9f9f-2d1e96792a87-openshift-service-ca\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.397705 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.397560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-maas-api-tls\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.397705 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.397591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-metrics-certs\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.397705 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.397639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7wz\" (UniqueName: \"kubernetes.io/projected/56edc13f-f855-4642-9f9f-2d1e96792a87-kube-api-access-rp7wz\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.398320 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.398296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56edc13f-f855-4642-9f9f-2d1e96792a87-openshift-service-ca\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.399906 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.399876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-maas-api-tls\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.399983 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.399964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-metrics-certs\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.405550 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.405532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7wz\" (UniqueName: \"kubernetes.io/projected/56edc13f-f855-4642-9f9f-2d1e96792a87-kube-api-access-rp7wz\") pod \"maas-api-e2e-ait-140d8435-6677498577-5nr7s\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.499316 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.499242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:08.768330 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.768303    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"]
Sep 15 19:09:08.772497 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:09:08.772467    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56edc13f_f855_4642_9f9f_2d1e96792a87.slice/crio-37d8c57fb51da8396028b81b37a530b9463728af2b4c5b5532737c5994a08f14 WatchSource:0}: Error finding container 37d8c57fb51da8396028b81b37a530b9463728af2b4c5b5532737c5994a08f14: Status 404 returned error can't find the container with id 37d8c57fb51da8396028b81b37a530b9463728af2b4c5b5532737c5994a08f14
Sep 15 19:09:08.872297 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.872265    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"]
Sep 15 19:09:08.876575 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.876558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:08.878966 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.878944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-140d8435-dockercfg-hrsmc\""
Sep 15 19:09:08.879082 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.878979    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-140d8435\""
Sep 15 19:09:08.885037 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.885012    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"]
Sep 15 19:09:08.895671 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.895648    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"]
Sep 15 19:09:08.898877 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.898857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:08.909758 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:08.909736    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"]
Sep 15 19:09:09.002125 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.002080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x86bl\" (UniqueName: \"kubernetes.io/projected/b364ff78-23cb-4170-84ce-da34a5ca3446-kube-api-access-x86bl\") pod \"payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr\" (UID: \"b364ff78-23cb-4170-84ce-da34a5ca3446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:09.002300 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.002131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a971179e-cc9c-46d1-afa4-2949e3ef157b-plugins-config-volume\") pod \"payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls\" (UID: \"a971179e-cc9c-46d1-afa4-2949e3ef157b\") " pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:09.002300 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.002255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9kzh\" (UniqueName: \"kubernetes.io/projected/a971179e-cc9c-46d1-afa4-2949e3ef157b-kube-api-access-c9kzh\") pod \"payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls\" (UID: \"a971179e-cc9c-46d1-afa4-2949e3ef157b\") " pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:09.002419 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.002334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b364ff78-23cb-4170-84ce-da34a5ca3446-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr\" (UID: \"b364ff78-23cb-4170-84ce-da34a5ca3446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:09.103535 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.103512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x86bl\" (UniqueName: \"kubernetes.io/projected/b364ff78-23cb-4170-84ce-da34a5ca3446-kube-api-access-x86bl\") pod \"payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr\" (UID: \"b364ff78-23cb-4170-84ce-da34a5ca3446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:09.103634 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.103552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a971179e-cc9c-46d1-afa4-2949e3ef157b-plugins-config-volume\") pod \"payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls\" (UID: \"a971179e-cc9c-46d1-afa4-2949e3ef157b\") " pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:09.103634 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.103596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9kzh\" (UniqueName: \"kubernetes.io/projected/a971179e-cc9c-46d1-afa4-2949e3ef157b-kube-api-access-c9kzh\") pod \"payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls\" (UID: \"a971179e-cc9c-46d1-afa4-2949e3ef157b\") " pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:09.103748 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.103644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b364ff78-23cb-4170-84ce-da34a5ca3446-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr\" (UID: \"b364ff78-23cb-4170-84ce-da34a5ca3446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:09.104272 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.104248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a971179e-cc9c-46d1-afa4-2949e3ef157b-plugins-config-volume\") pod \"payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls\" (UID: \"a971179e-cc9c-46d1-afa4-2949e3ef157b\") " pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:09.104391 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.104371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b364ff78-23cb-4170-84ce-da34a5ca3446-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr\" (UID: \"b364ff78-23cb-4170-84ce-da34a5ca3446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:09.111098 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.111077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9kzh\" (UniqueName: \"kubernetes.io/projected/a971179e-cc9c-46d1-afa4-2949e3ef157b-kube-api-access-c9kzh\") pod \"payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls\" (UID: \"a971179e-cc9c-46d1-afa4-2949e3ef157b\") " pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:09.111179 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.111149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x86bl\" (UniqueName: \"kubernetes.io/projected/b364ff78-23cb-4170-84ce-da34a5ca3446-kube-api-access-x86bl\") pod \"payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr\" (UID: \"b364ff78-23cb-4170-84ce-da34a5ca3446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:09.188788 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.188763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:09.207726 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.207702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:09.276731 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.276697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s" event={"ID":"56edc13f-f855-4642-9f9f-2d1e96792a87","Type":"ContainerStarted","Data":"e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae"}
Sep 15 19:09:09.276875 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.276752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s" event={"ID":"56edc13f-f855-4642-9f9f-2d1e96792a87","Type":"ContainerStarted","Data":"37d8c57fb51da8396028b81b37a530b9463728af2b4c5b5532737c5994a08f14"}
Sep 15 19:09:09.277038 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.276993    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:09.293723 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.293670    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s" podStartSLOduration=0.970809321 podStartE2EDuration="1.293651539s" totalImagesPullingTime="322.842218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:09.292862323 +0000 UTC m=+1099.834957855" watchObservedRunningTime="2026-09-15 19:09:09.293651539 +0000 UTC m=+1099.835747068"
Sep 15 19:09:09.462607 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.458272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"]
Sep 15 19:09:09.464219 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:09:09.464181    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb364ff78_23cb_4170_84ce_da34a5ca3446.slice/crio-c9b6ef506bbf47963b415736ffa492d4584a90ff9824bf2b71706c41bd7d54c6 WatchSource:0}: Error finding container c9b6ef506bbf47963b415736ffa492d4584a90ff9824bf2b71706c41bd7d54c6: Status 404 returned error can't find the container with id c9b6ef506bbf47963b415736ffa492d4584a90ff9824bf2b71706c41bd7d54c6
Sep 15 19:09:09.483865 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:09.483844    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"]
Sep 15 19:09:09.485773 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:09:09.485743    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda971179e_cc9c_46d1_afa4_2949e3ef157b.slice/crio-15683469cc41e2073499dd420bc7cbb24967451805b4a121c2da37a4ceff361c WatchSource:0}: Error finding container 15683469cc41e2073499dd420bc7cbb24967451805b4a121c2da37a4ceff361c: Status 404 returned error can't find the container with id 15683469cc41e2073499dd420bc7cbb24967451805b4a121c2da37a4ceff361c
Sep 15 19:09:10.282623 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:10.282590    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr" event={"ID":"b364ff78-23cb-4170-84ce-da34a5ca3446","Type":"ContainerStarted","Data":"66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66"}
Sep 15 19:09:10.283058 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:10.282641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr" event={"ID":"b364ff78-23cb-4170-84ce-da34a5ca3446","Type":"ContainerStarted","Data":"c9b6ef506bbf47963b415736ffa492d4584a90ff9824bf2b71706c41bd7d54c6"}
Sep 15 19:09:10.283058 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:10.282683    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:10.284173 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:10.284141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls" event={"ID":"a971179e-cc9c-46d1-afa4-2949e3ef157b","Type":"ContainerStarted","Data":"5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094"}
Sep 15 19:09:10.284306 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:10.284189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls" event={"ID":"a971179e-cc9c-46d1-afa4-2949e3ef157b","Type":"ContainerStarted","Data":"15683469cc41e2073499dd420bc7cbb24967451805b4a121c2da37a4ceff361c"}
Sep 15 19:09:10.284375 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:10.284319    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:10.303618 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:10.303560    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr" podStartSLOduration=2.303543467 podStartE2EDuration="2.303543467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:10.29966341 +0000 UTC m=+1100.841758939" watchObservedRunningTime="2026-09-15 19:09:10.303543467 +0000 UTC m=+1100.845638999"
Sep 15 19:09:10.319567 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:10.319518    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls" podStartSLOduration=2.319506143 podStartE2EDuration="2.319506143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:10.318466729 +0000 UTC m=+1100.860562259" watchObservedRunningTime="2026-09-15 19:09:10.319506143 +0000 UTC m=+1100.861601671"
Sep 15 19:09:15.289261 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:15.289233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:21.289971 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:21.289939    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:21.290394 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:21.290013    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:26.381123 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:26.381083    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:09:37.010782 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.010751    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"]
Sep 15 19:09:37.011316 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.010963    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s" podUID="56edc13f-f855-4642-9f9f-2d1e96792a87" containerName="maas-api" containerID="cri-o://e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae" gracePeriod=30
Sep 15 19:09:37.068416 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.068384    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"]
Sep 15 19:09:37.068652 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.068624    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls" podUID="a971179e-cc9c-46d1-afa4-2949e3ef157b" containerName="payload-processing" containerID="cri-o://5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094" gracePeriod=30
Sep 15 19:09:37.074181 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.074152    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"]
Sep 15 19:09:37.074443 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.074416    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr" podUID="b364ff78-23cb-4170-84ce-da34a5ca3446" containerName="payload-pre-processing" containerID="cri-o://66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66" gracePeriod=30
Sep 15 19:09:37.337256 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.337230    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:37.340744 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.340722    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:37.346606 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.346583    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:37.408604 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.408570    2578 generic.go:356] "Generic (PLEG): container finished" podID="56edc13f-f855-4642-9f9f-2d1e96792a87" containerID="e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae" exitCode=0
Sep 15 19:09:37.408763 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.408617    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"
Sep 15 19:09:37.408763 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.408652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s" event={"ID":"56edc13f-f855-4642-9f9f-2d1e96792a87","Type":"ContainerDied","Data":"e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae"}
Sep 15 19:09:37.408763 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.408710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s" event={"ID":"56edc13f-f855-4642-9f9f-2d1e96792a87","Type":"ContainerDied","Data":"37d8c57fb51da8396028b81b37a530b9463728af2b4c5b5532737c5994a08f14"}
Sep 15 19:09:37.408763 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.408736    2578 scope.go:117] "RemoveContainer" containerID="e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae"
Sep 15 19:09:37.409890 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.409871    2578 generic.go:356] "Generic (PLEG): container finished" podID="b364ff78-23cb-4170-84ce-da34a5ca3446" containerID="66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66" exitCode=0
Sep 15 19:09:37.409963 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.409927    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"
Sep 15 19:09:37.409963 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.409953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr" event={"ID":"b364ff78-23cb-4170-84ce-da34a5ca3446","Type":"ContainerDied","Data":"66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66"}
Sep 15 19:09:37.410049 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.409988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr" event={"ID":"b364ff78-23cb-4170-84ce-da34a5ca3446","Type":"ContainerDied","Data":"c9b6ef506bbf47963b415736ffa492d4584a90ff9824bf2b71706c41bd7d54c6"}
Sep 15 19:09:37.411171 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.411151    2578 generic.go:356] "Generic (PLEG): container finished" podID="a971179e-cc9c-46d1-afa4-2949e3ef157b" containerID="5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094" exitCode=0
Sep 15 19:09:37.411260 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.411180    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"
Sep 15 19:09:37.411298 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.411178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls" event={"ID":"a971179e-cc9c-46d1-afa4-2949e3ef157b","Type":"ContainerDied","Data":"5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094"}
Sep 15 19:09:37.411298 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.411287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls" event={"ID":"a971179e-cc9c-46d1-afa4-2949e3ef157b","Type":"ContainerDied","Data":"15683469cc41e2073499dd420bc7cbb24967451805b4a121c2da37a4ceff361c"}
Sep 15 19:09:37.417472 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.417456    2578 scope.go:117] "RemoveContainer" containerID="e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae"
Sep 15 19:09:37.417745 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:09:37.417726    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae\": container with ID starting with e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae not found: ID does not exist" containerID="e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae"
Sep 15 19:09:37.417831 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.417751    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae"} err="failed to get container status \"e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae\": rpc error: code = NotFound desc = could not find container \"e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae\": container with ID starting with e402108190bd58aec51b89b01aee107a48b270bf8c366cb3443401e9271ec5ae not found: ID does not exist"
Sep 15 19:09:37.417831 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.417767    2578 scope.go:117] "RemoveContainer" containerID="66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66"
Sep 15 19:09:37.425112 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.425094    2578 scope.go:117] "RemoveContainer" containerID="66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66"
Sep 15 19:09:37.425356 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:09:37.425337    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66\": container with ID starting with 66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66 not found: ID does not exist" containerID="66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66"
Sep 15 19:09:37.425429 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.425367    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66"} err="failed to get container status \"66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66\": rpc error: code = NotFound desc = could not find container \"66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66\": container with ID starting with 66672f63637231a5ce94ece066671c22b3cd084c92b496d59eeecd37b8b8ff66 not found: ID does not exist"
Sep 15 19:09:37.425429 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.425389    2578 scope.go:117] "RemoveContainer" containerID="5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094"
Sep 15 19:09:37.432315 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.432298    2578 scope.go:117] "RemoveContainer" containerID="5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094"
Sep 15 19:09:37.432550 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:09:37.432535    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094\": container with ID starting with 5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094 not found: ID does not exist" containerID="5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094"
Sep 15 19:09:37.432607 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.432556    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094"} err="failed to get container status \"5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094\": rpc error: code = NotFound desc = could not find container \"5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094\": container with ID starting with 5a79961ea8c021be5a3275f229d32cca2e8eb24d831e9a713b0de49bac207094 not found: ID does not exist"
Sep 15 19:09:37.440891 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.440876    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a971179e-cc9c-46d1-afa4-2949e3ef157b-plugins-config-volume\") pod \"a971179e-cc9c-46d1-afa4-2949e3ef157b\" (UID: \"a971179e-cc9c-46d1-afa4-2949e3ef157b\") "
Sep 15 19:09:37.440973 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.440900    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x86bl\" (UniqueName: \"kubernetes.io/projected/b364ff78-23cb-4170-84ce-da34a5ca3446-kube-api-access-x86bl\") pod \"b364ff78-23cb-4170-84ce-da34a5ca3446\" (UID: \"b364ff78-23cb-4170-84ce-da34a5ca3446\") "
Sep 15 19:09:37.440973 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.440927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9kzh\" (UniqueName: \"kubernetes.io/projected/a971179e-cc9c-46d1-afa4-2949e3ef157b-kube-api-access-c9kzh\") pod \"a971179e-cc9c-46d1-afa4-2949e3ef157b\" (UID: \"a971179e-cc9c-46d1-afa4-2949e3ef157b\") "
Sep 15 19:09:37.440973 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.440950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-maas-api-tls\") pod \"56edc13f-f855-4642-9f9f-2d1e96792a87\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") "
Sep 15 19:09:37.441126 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.441022    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56edc13f-f855-4642-9f9f-2d1e96792a87-openshift-service-ca\") pod \"56edc13f-f855-4642-9f9f-2d1e96792a87\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") "
Sep 15 19:09:37.441126 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.441056    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp7wz\" (UniqueName: \"kubernetes.io/projected/56edc13f-f855-4642-9f9f-2d1e96792a87-kube-api-access-rp7wz\") pod \"56edc13f-f855-4642-9f9f-2d1e96792a87\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") "
Sep 15 19:09:37.441126 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.441097    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b364ff78-23cb-4170-84ce-da34a5ca3446-plugins-config-volume\") pod \"b364ff78-23cb-4170-84ce-da34a5ca3446\" (UID: \"b364ff78-23cb-4170-84ce-da34a5ca3446\") "
Sep 15 19:09:37.441284 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.441130    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-metrics-certs\") pod \"56edc13f-f855-4642-9f9f-2d1e96792a87\" (UID: \"56edc13f-f855-4642-9f9f-2d1e96792a87\") "
Sep 15 19:09:37.441396 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.441370    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a971179e-cc9c-46d1-afa4-2949e3ef157b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a971179e-cc9c-46d1-afa4-2949e3ef157b" (UID: "a971179e-cc9c-46d1-afa4-2949e3ef157b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:37.441575 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.441530    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56edc13f-f855-4642-9f9f-2d1e96792a87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56edc13f-f855-4642-9f9f-2d1e96792a87" (UID: "56edc13f-f855-4642-9f9f-2d1e96792a87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:37.441753 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.441721    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b364ff78-23cb-4170-84ce-da34a5ca3446-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b364ff78-23cb-4170-84ce-da34a5ca3446" (UID: "b364ff78-23cb-4170-84ce-da34a5ca3446"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:37.443130 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.443091    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56edc13f-f855-4642-9f9f-2d1e96792a87" (UID: "56edc13f-f855-4642-9f9f-2d1e96792a87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:37.443223 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.443162    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b364ff78-23cb-4170-84ce-da34a5ca3446-kube-api-access-x86bl" (OuterVolumeSpecName: "kube-api-access-x86bl") pod "b364ff78-23cb-4170-84ce-da34a5ca3446" (UID: "b364ff78-23cb-4170-84ce-da34a5ca3446"). InnerVolumeSpecName "kube-api-access-x86bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:37.443223 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.443193    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a971179e-cc9c-46d1-afa4-2949e3ef157b-kube-api-access-c9kzh" (OuterVolumeSpecName: "kube-api-access-c9kzh") pod "a971179e-cc9c-46d1-afa4-2949e3ef157b" (UID: "a971179e-cc9c-46d1-afa4-2949e3ef157b"). InnerVolumeSpecName "kube-api-access-c9kzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:37.443604 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.443586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "56edc13f-f855-4642-9f9f-2d1e96792a87" (UID: "56edc13f-f855-4642-9f9f-2d1e96792a87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:09:37.443604 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.443595    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56edc13f-f855-4642-9f9f-2d1e96792a87-kube-api-access-rp7wz" (OuterVolumeSpecName: "kube-api-access-rp7wz") pod "56edc13f-f855-4642-9f9f-2d1e96792a87" (UID: "56edc13f-f855-4642-9f9f-2d1e96792a87"). InnerVolumeSpecName "kube-api-access-rp7wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:37.542730 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.542683    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x86bl\" (UniqueName: \"kubernetes.io/projected/b364ff78-23cb-4170-84ce-da34a5ca3446-kube-api-access-x86bl\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:37.542730 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.542721    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9kzh\" (UniqueName: \"kubernetes.io/projected/a971179e-cc9c-46d1-afa4-2949e3ef157b-kube-api-access-c9kzh\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:37.542730 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.542735    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:37.542993 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.542747    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56edc13f-f855-4642-9f9f-2d1e96792a87-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:37.542993 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.542759    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp7wz\" (UniqueName: \"kubernetes.io/projected/56edc13f-f855-4642-9f9f-2d1e96792a87-kube-api-access-rp7wz\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:37.542993 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.542771    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b364ff78-23cb-4170-84ce-da34a5ca3446-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:37.542993 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.542783    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56edc13f-f855-4642-9f9f-2d1e96792a87-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:37.542993 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.542794    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a971179e-cc9c-46d1-afa4-2949e3ef157b-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:09:37.733479 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.733454    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"]
Sep 15 19:09:37.737325 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.737303    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-140d8435-6677498577-5nr7s"]
Sep 15 19:09:37.746366 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.746341    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"]
Sep 15 19:09:37.749176 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.749153    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-140d8435-64f5cbcc57-9tsls"]
Sep 15 19:09:37.758863 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.758835    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"]
Sep 15 19:09:37.762329 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:37.762309    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-140d8435-66648d4786-sdvnr"]
Sep 15 19:09:38.098010 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:38.097977    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56edc13f-f855-4642-9f9f-2d1e96792a87" path="/var/lib/kubelet/pods/56edc13f-f855-4642-9f9f-2d1e96792a87/volumes"
Sep 15 19:09:38.098384 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:38.098345    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a971179e-cc9c-46d1-afa4-2949e3ef157b" path="/var/lib/kubelet/pods/a971179e-cc9c-46d1-afa4-2949e3ef157b/volumes"
Sep 15 19:09:38.098649 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:38.098637    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b364ff78-23cb-4170-84ce-da34a5ca3446" path="/var/lib/kubelet/pods/b364ff78-23cb-4170-84ce-da34a5ca3446/volumes"
Sep 15 19:09:43.021427 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:43.021378    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:09:44.491448 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.491413    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"]
Sep 15 19:09:44.491829 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.491755    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b364ff78-23cb-4170-84ce-da34a5ca3446" containerName="payload-pre-processing"
Sep 15 19:09:44.491829 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.491767    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b364ff78-23cb-4170-84ce-da34a5ca3446" containerName="payload-pre-processing"
Sep 15 19:09:44.491829 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.491783    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56edc13f-f855-4642-9f9f-2d1e96792a87" containerName="maas-api"
Sep 15 19:09:44.491829 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.491788    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="56edc13f-f855-4642-9f9f-2d1e96792a87" containerName="maas-api"
Sep 15 19:09:44.491829 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.491808    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a971179e-cc9c-46d1-afa4-2949e3ef157b" containerName="payload-processing"
Sep 15 19:09:44.491829 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.491815    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a971179e-cc9c-46d1-afa4-2949e3ef157b" containerName="payload-processing"
Sep 15 19:09:44.492015 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.491863    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a971179e-cc9c-46d1-afa4-2949e3ef157b" containerName="payload-processing"
Sep 15 19:09:44.492015 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.491871    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="56edc13f-f855-4642-9f9f-2d1e96792a87" containerName="maas-api"
Sep 15 19:09:44.492015 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.491880    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b364ff78-23cb-4170-84ce-da34a5ca3446" containerName="payload-pre-processing"
Sep 15 19:09:44.496349 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.496328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.498830 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.498809    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-282a6587-serving-cert\""
Sep 15 19:09:44.508045 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.508019    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"]
Sep 15 19:09:44.603028 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.602991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4dcz\" (UniqueName: \"kubernetes.io/projected/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-kube-api-access-b4dcz\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.603246 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.603036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-metrics-certs\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.603246 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.603127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-openshift-service-ca\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.603246 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.603165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-maas-api-tls\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.704020 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.703984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-metrics-certs\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.704213 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.704029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-openshift-service-ca\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.704213 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.704058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-maas-api-tls\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.704213 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.704148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4dcz\" (UniqueName: \"kubernetes.io/projected/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-kube-api-access-b4dcz\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.704333 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:09:44.704215    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-serving-cert: secret "maas-api-e2e-derive-282a6587-serving-cert" not found
Sep 15 19:09:44.704333 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:09:44.704315    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-maas-api-tls podName:ff64c90f-35e1-48fd-bd23-2c895f7e06b0 nodeName:}" failed. No retries permitted until 2026-09-15 19:09:45.204297326 +0000 UTC m=+1135.746392833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-maas-api-tls") pod "maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx" (UID: "ff64c90f-35e1-48fd-bd23-2c895f7e06b0") : secret "maas-api-e2e-derive-282a6587-serving-cert" not found
Sep 15 19:09:44.704779 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.704760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-openshift-service-ca\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.706413 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.706396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-metrics-certs\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:44.714872 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:44.714848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4dcz\" (UniqueName: \"kubernetes.io/projected/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-kube-api-access-b4dcz\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:45.201880 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.201844    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"]
Sep 15 19:09:45.205453 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.205437    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:09:45.208250 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.208224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-maas-api-tls\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:45.208775 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.208756    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-282a6587\""
Sep 15 19:09:45.208968 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.208948    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-282a6587-dockercfg-ndzdt\""
Sep 15 19:09:45.210786 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.210765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-maas-api-tls\") pod \"maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:45.217979 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.217952    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"]
Sep 15 19:09:45.309054 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.309018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jsz\" (UniqueName: \"kubernetes.io/projected/f177aa08-4675-43bd-bf80-991bd0f04ee4-kube-api-access-j2jsz\") pod \"payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb\" (UID: \"f177aa08-4675-43bd-bf80-991bd0f04ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:09:45.309202 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.309158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f177aa08-4675-43bd-bf80-991bd0f04ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb\" (UID: \"f177aa08-4675-43bd-bf80-991bd0f04ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:09:45.407001 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.406964    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:45.409661 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.409632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jsz\" (UniqueName: \"kubernetes.io/projected/f177aa08-4675-43bd-bf80-991bd0f04ee4-kube-api-access-j2jsz\") pod \"payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb\" (UID: \"f177aa08-4675-43bd-bf80-991bd0f04ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:09:45.409792 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.409713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f177aa08-4675-43bd-bf80-991bd0f04ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb\" (UID: \"f177aa08-4675-43bd-bf80-991bd0f04ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:09:45.410409 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.410387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f177aa08-4675-43bd-bf80-991bd0f04ee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb\" (UID: \"f177aa08-4675-43bd-bf80-991bd0f04ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:09:45.417817 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.417782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jsz\" (UniqueName: \"kubernetes.io/projected/f177aa08-4675-43bd-bf80-991bd0f04ee4-kube-api-access-j2jsz\") pod \"payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb\" (UID: \"f177aa08-4675-43bd-bf80-991bd0f04ee4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:09:45.524572 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.524540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:09:45.677572 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.677511    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"]
Sep 15 19:09:45.679820 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:09:45.679784    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff64c90f_35e1_48fd_bd23_2c895f7e06b0.slice/crio-2908cbc3325e10822adb4e5af0f60b6b31f1a5d31887a061b681034936352138 WatchSource:0}: Error finding container 2908cbc3325e10822adb4e5af0f60b6b31f1a5d31887a061b681034936352138: Status 404 returned error can't find the container with id 2908cbc3325e10822adb4e5af0f60b6b31f1a5d31887a061b681034936352138
Sep 15 19:09:45.794662 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:45.794637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"]
Sep 15 19:09:45.796945 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:09:45.796910    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf177aa08_4675_43bd_bf80_991bd0f04ee4.slice/crio-f0bf1a1aa745448662077b79760175592587ade9f4a2b49b5e1a737a16cecb83 WatchSource:0}: Error finding container f0bf1a1aa745448662077b79760175592587ade9f4a2b49b5e1a737a16cecb83: Status 404 returned error can't find the container with id f0bf1a1aa745448662077b79760175592587ade9f4a2b49b5e1a737a16cecb83
Sep 15 19:09:46.455478 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:46.455451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb" event={"ID":"f177aa08-4675-43bd-bf80-991bd0f04ee4","Type":"ContainerStarted","Data":"0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99"}
Sep 15 19:09:46.455629 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:46.455499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb" event={"ID":"f177aa08-4675-43bd-bf80-991bd0f04ee4","Type":"ContainerStarted","Data":"f0bf1a1aa745448662077b79760175592587ade9f4a2b49b5e1a737a16cecb83"}
Sep 15 19:09:46.455629 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:46.455555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:09:46.456710 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:46.456690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx" event={"ID":"ff64c90f-35e1-48fd-bd23-2c895f7e06b0","Type":"ContainerStarted","Data":"2908cbc3325e10822adb4e5af0f60b6b31f1a5d31887a061b681034936352138"}
Sep 15 19:09:46.470742 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:46.470694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb" podStartSLOduration=1.470679182 podStartE2EDuration="1.470679182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:46.469271478 +0000 UTC m=+1137.011367008" watchObservedRunningTime="2026-09-15 19:09:46.470679182 +0000 UTC m=+1137.012774711"
Sep 15 19:09:47.463432 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:47.463397    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx" event={"ID":"ff64c90f-35e1-48fd-bd23-2c895f7e06b0","Type":"ContainerStarted","Data":"868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec"}
Sep 15 19:09:47.479962 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:47.479914    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx" podStartSLOduration=2.755260958 podStartE2EDuration="3.479894798s" totalImagesPullingTime="724.63384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:47.478851834 +0000 UTC m=+1138.020947360" watchObservedRunningTime="2026-09-15 19:09:47.479894798 +0000 UTC m=+1138.021990320"
Sep 15 19:09:48.468449 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:48.468409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:54.477762 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:54.477728    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:09:57.465542 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:09:57.465517    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:10:01.881429 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:01.881391    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:10:23.226145 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.226060    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"]
Sep 15 19:10:23.226586 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.226285    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx" podUID="ff64c90f-35e1-48fd-bd23-2c895f7e06b0" containerName="maas-api" containerID="cri-o://868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec" gracePeriod=30
Sep 15 19:10:23.286054 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.286021    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"]
Sep 15 19:10:23.286277 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.286235    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb" podUID="f177aa08-4675-43bd-bf80-991bd0f04ee4" containerName="payload-pre-processing" containerID="cri-o://0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99" gracePeriod=30
Sep 15 19:10:23.542667 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.542641    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:10:23.563217 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.563163    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:10:23.624228 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.624195    2578 generic.go:356] "Generic (PLEG): container finished" podID="f177aa08-4675-43bd-bf80-991bd0f04ee4" containerID="0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99" exitCode=0
Sep 15 19:10:23.624407 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.624258    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"
Sep 15 19:10:23.624407 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.624275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb" event={"ID":"f177aa08-4675-43bd-bf80-991bd0f04ee4","Type":"ContainerDied","Data":"0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99"}
Sep 15 19:10:23.624407 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.624336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb" event={"ID":"f177aa08-4675-43bd-bf80-991bd0f04ee4","Type":"ContainerDied","Data":"f0bf1a1aa745448662077b79760175592587ade9f4a2b49b5e1a737a16cecb83"}
Sep 15 19:10:23.624407 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.624373    2578 scope.go:117] "RemoveContainer" containerID="0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99"
Sep 15 19:10:23.625492 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625355    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4dcz\" (UniqueName: \"kubernetes.io/projected/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-kube-api-access-b4dcz\") pod \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") "
Sep 15 19:10:23.625492 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625431    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-maas-api-tls\") pod \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") "
Sep 15 19:10:23.625492 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-openshift-service-ca\") pod \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") "
Sep 15 19:10:23.625492 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625488    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2jsz\" (UniqueName: \"kubernetes.io/projected/f177aa08-4675-43bd-bf80-991bd0f04ee4-kube-api-access-j2jsz\") pod \"f177aa08-4675-43bd-bf80-991bd0f04ee4\" (UID: \"f177aa08-4675-43bd-bf80-991bd0f04ee4\") "
Sep 15 19:10:23.625720 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625512    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f177aa08-4675-43bd-bf80-991bd0f04ee4-plugins-config-volume\") pod \"f177aa08-4675-43bd-bf80-991bd0f04ee4\" (UID: \"f177aa08-4675-43bd-bf80-991bd0f04ee4\") "
Sep 15 19:10:23.625720 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625544    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-metrics-certs\") pod \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\" (UID: \"ff64c90f-35e1-48fd-bd23-2c895f7e06b0\") "
Sep 15 19:10:23.625720 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625696    2578 generic.go:356] "Generic (PLEG): container finished" podID="ff64c90f-35e1-48fd-bd23-2c895f7e06b0" containerID="868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec" exitCode=0
Sep 15 19:10:23.625909 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx" event={"ID":"ff64c90f-35e1-48fd-bd23-2c895f7e06b0","Type":"ContainerDied","Data":"868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec"}
Sep 15 19:10:23.625909 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx" event={"ID":"ff64c90f-35e1-48fd-bd23-2c895f7e06b0","Type":"ContainerDied","Data":"2908cbc3325e10822adb4e5af0f60b6b31f1a5d31887a061b681034936352138"}
Sep 15 19:10:23.625909 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625815    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff64c90f-35e1-48fd-bd23-2c895f7e06b0" (UID: "ff64c90f-35e1-48fd-bd23-2c895f7e06b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:10:23.625909 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.625872    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"
Sep 15 19:10:23.626348 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.626315    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f177aa08-4675-43bd-bf80-991bd0f04ee4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f177aa08-4675-43bd-bf80-991bd0f04ee4" (UID: "f177aa08-4675-43bd-bf80-991bd0f04ee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:10:23.627701 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.627652    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff64c90f-35e1-48fd-bd23-2c895f7e06b0" (UID: "ff64c90f-35e1-48fd-bd23-2c895f7e06b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:10:23.628156 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.628134    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-kube-api-access-b4dcz" (OuterVolumeSpecName: "kube-api-access-b4dcz") pod "ff64c90f-35e1-48fd-bd23-2c895f7e06b0" (UID: "ff64c90f-35e1-48fd-bd23-2c895f7e06b0"). InnerVolumeSpecName "kube-api-access-b4dcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:10:23.628259 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.628176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ff64c90f-35e1-48fd-bd23-2c895f7e06b0" (UID: "ff64c90f-35e1-48fd-bd23-2c895f7e06b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:10:23.628319 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.628304    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f177aa08-4675-43bd-bf80-991bd0f04ee4-kube-api-access-j2jsz" (OuterVolumeSpecName: "kube-api-access-j2jsz") pod "f177aa08-4675-43bd-bf80-991bd0f04ee4" (UID: "f177aa08-4675-43bd-bf80-991bd0f04ee4"). InnerVolumeSpecName "kube-api-access-j2jsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:10:23.633074 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.633056    2578 scope.go:117] "RemoveContainer" containerID="0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99"
Sep 15 19:10:23.633334 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:10:23.633314    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99\": container with ID starting with 0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99 not found: ID does not exist" containerID="0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99"
Sep 15 19:10:23.633410 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.633346    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99"} err="failed to get container status \"0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99\": rpc error: code = NotFound desc = could not find container \"0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99\": container with ID starting with 0ab531751ff22ba5d012cbfe905a99e3e97c35ae877b42a4148821649cc00e99 not found: ID does not exist"
Sep 15 19:10:23.633410 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.633368    2578 scope.go:117] "RemoveContainer" containerID="868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec"
Sep 15 19:10:23.640946 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.640929    2578 scope.go:117] "RemoveContainer" containerID="868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec"
Sep 15 19:10:23.641201 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:10:23.641184    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec\": container with ID starting with 868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec not found: ID does not exist" containerID="868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec"
Sep 15 19:10:23.641256 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.641210    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec"} err="failed to get container status \"868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec\": rpc error: code = NotFound desc = could not find container \"868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec\": container with ID starting with 868772d34f7f80debf9c6f93d75a585e02190e824046a9b5a8e65cad43eeb7ec not found: ID does not exist"
Sep 15 19:10:23.726915 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.726875    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:10:23.726915 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.726916    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jsz\" (UniqueName: \"kubernetes.io/projected/f177aa08-4675-43bd-bf80-991bd0f04ee4-kube-api-access-j2jsz\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:10:23.727121 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.726932    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f177aa08-4675-43bd-bf80-991bd0f04ee4-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:10:23.727121 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.726946    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:10:23.727121 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.726960    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4dcz\" (UniqueName: \"kubernetes.io/projected/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-kube-api-access-b4dcz\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:10:23.727121 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.726973    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff64c90f-35e1-48fd-bd23-2c895f7e06b0-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:10:23.948216 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.948189    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"]
Sep 15 19:10:23.951746 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.951722    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-282a6587-7868594cd6-gd5zb"]
Sep 15 19:10:23.964622 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.964601    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"]
Sep 15 19:10:23.970573 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:23.970551    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-282a6587-77cd4cc9b8-hlkqx"]
Sep 15 19:10:24.098129 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:24.098096    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f177aa08-4675-43bd-bf80-991bd0f04ee4" path="/var/lib/kubelet/pods/f177aa08-4675-43bd-bf80-991bd0f04ee4/volumes"
Sep 15 19:10:24.098481 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:24.098469    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff64c90f-35e1-48fd-bd23-2c895f7e06b0" path="/var/lib/kubelet/pods/ff64c90f-35e1-48fd-bd23-2c895f7e06b0/volumes"
Sep 15 19:10:28.619838 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:28.619788    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:10:45.373447 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:10:45.373409    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:11:08.592207 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:08.592170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:11:23.202340 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.202281    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"]
Sep 15 19:11:23.202786 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.202759    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f177aa08-4675-43bd-bf80-991bd0f04ee4" containerName="payload-pre-processing"
Sep 15 19:11:23.202786 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.202779    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f177aa08-4675-43bd-bf80-991bd0f04ee4" containerName="payload-pre-processing"
Sep 15 19:11:23.202927 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.202831    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff64c90f-35e1-48fd-bd23-2c895f7e06b0" containerName="maas-api"
Sep 15 19:11:23.202927 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.202840    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff64c90f-35e1-48fd-bd23-2c895f7e06b0" containerName="maas-api"
Sep 15 19:11:23.203027 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.202930    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f177aa08-4675-43bd-bf80-991bd0f04ee4" containerName="payload-pre-processing"
Sep 15 19:11:23.203027 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.202948    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff64c90f-35e1-48fd-bd23-2c895f7e06b0" containerName="maas-api"
Sep 15 19:11:23.207281 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.207255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.211573 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.211548    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3daa58\"/\"test-model-3daa58-kserve-self-signed-certs\""
Sep 15 19:11:23.211711 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.211567    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3daa58\"/\"default-dockercfg-dgxrh\""
Sep 15 19:11:23.211711 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.211567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3daa58\"/\"openshift-service-ca.crt\""
Sep 15 19:11:23.211711 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.211579    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3daa58\"/\"kube-root-ca.crt\""
Sep 15 19:11:23.224399 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.224376    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"]
Sep 15 19:11:23.380667 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.380637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8sq8\" (UniqueName: \"kubernetes.io/projected/c399bda2-751f-41db-a9e8-094f54a104cd-kube-api-access-p8sq8\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.380849 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.380676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-model-cache\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.380849 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.380702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-dshm\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.380849 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.380741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-home\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.380849 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.380762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-tmp-dir\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.381007 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.380849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c399bda2-751f-41db-a9e8-094f54a104cd-tls-certs\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.481348 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.481264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-model-cache\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.481348 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.481304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-dshm\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.481348 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.481326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-home\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.481348 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.481350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-tmp-dir\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.481596 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.481376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c399bda2-751f-41db-a9e8-094f54a104cd-tls-certs\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.481596 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.481422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8sq8\" (UniqueName: \"kubernetes.io/projected/c399bda2-751f-41db-a9e8-094f54a104cd-kube-api-access-p8sq8\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.481733 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.481701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-model-cache\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.481831 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.481719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-home\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.481831 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.481819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-tmp-dir\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.483630 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.483610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-dshm\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.483933 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.483918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c399bda2-751f-41db-a9e8-094f54a104cd-tls-certs\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.489058 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.489037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8sq8\" (UniqueName: \"kubernetes.io/projected/c399bda2-751f-41db-a9e8-094f54a104cd-kube-api-access-p8sq8\") pod \"test-model-3daa58-kserve-7c4684df65-fhm5j\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") " pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.517192 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.517155    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:23.785948 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.785922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"]
Sep 15 19:11:23.788277 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:11:23.788244    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc399bda2_751f_41db_a9e8_094f54a104cd.slice/crio-697b52c42936eb4cf0b22ae815ee7037c5fdefb2e5ab95aeb6248deb6515c963 WatchSource:0}: Error finding container 697b52c42936eb4cf0b22ae815ee7037c5fdefb2e5ab95aeb6248deb6515c963: Status 404 returned error can't find the container with id 697b52c42936eb4cf0b22ae815ee7037c5fdefb2e5ab95aeb6248deb6515c963
Sep 15 19:11:23.894524 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.894489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j" event={"ID":"c399bda2-751f-41db-a9e8-094f54a104cd","Type":"ContainerStarted","Data":"644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf"}
Sep 15 19:11:23.894688 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.894532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j" event={"ID":"c399bda2-751f-41db-a9e8-094f54a104cd","Type":"ContainerStarted","Data":"697b52c42936eb4cf0b22ae815ee7037c5fdefb2e5ab95aeb6248deb6515c963"}
Sep 15 19:11:23.911148 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:23.911099    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j" podStartSLOduration=0.911083327 podStartE2EDuration="911.083327ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:11:23.908996268 +0000 UTC m=+1234.451091796" watchObservedRunningTime="2026-09-15 19:11:23.911083327 +0000 UTC m=+1234.453178893"
Sep 15 19:11:32.577054 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:32.577018    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:11:33.518142 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:33.518108    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:33.518338 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:33.518152    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:33.530842 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:33.530818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:11:33.948362 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:11:33.948336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:12:00.687360 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:00.687276    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:12:06.633191 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.633153    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"]
Sep 15 19:12:06.636902 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.636851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.639187 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.639165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a0497c3b-serving-cert\""
Sep 15 19:12:06.645554 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.645530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"]
Sep 15 19:12:06.719618 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.719590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a49f0b2-bb8e-4265-8dbf-e16732793637-openshift-service-ca\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.719766 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.719626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-maas-api-tls\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.719766 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.719647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkp6m\" (UniqueName: \"kubernetes.io/projected/6a49f0b2-bb8e-4265-8dbf-e16732793637-kube-api-access-qkp6m\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.719766 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.719699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-metrics-certs\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.825486 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.825447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a49f0b2-bb8e-4265-8dbf-e16732793637-openshift-service-ca\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.825638 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.825525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-maas-api-tls\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.825638 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.825560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkp6m\" (UniqueName: \"kubernetes.io/projected/6a49f0b2-bb8e-4265-8dbf-e16732793637-kube-api-access-qkp6m\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.825638 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.825596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-metrics-certs\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.826219 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.826186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a49f0b2-bb8e-4265-8dbf-e16732793637-openshift-service-ca\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.828317 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.828295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-metrics-certs\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.828426 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.828368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-maas-api-tls\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.834254 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.834229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkp6m\" (UniqueName: \"kubernetes.io/projected/6a49f0b2-bb8e-4265-8dbf-e16732793637-kube-api-access-qkp6m\") pod \"maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:06.948340 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:06.948265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:07.211010 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.210943    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"]
Sep 15 19:12:07.214462 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.214444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:07.216603 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.216574    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a0497c3b\""
Sep 15 19:12:07.216767 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.216749    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a0497c3b-dockercfg-nxxkl\""
Sep 15 19:12:07.219640 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.219617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"]
Sep 15 19:12:07.222948 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.222922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"]
Sep 15 19:12:07.223373 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:12:07.223345    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a49f0b2_bb8e_4265_8dbf_e16732793637.slice/crio-907fde54236c29e5cd1c7c6c91d446b999a4be8fe1f1b55c0574259713df9860 WatchSource:0}: Error finding container 907fde54236c29e5cd1c7c6c91d446b999a4be8fe1f1b55c0574259713df9860: Status 404 returned error can't find the container with id 907fde54236c29e5cd1c7c6c91d446b999a4be8fe1f1b55c0574259713df9860
Sep 15 19:12:07.255825 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.255784    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"]
Sep 15 19:12:07.259013 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.258997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:07.269229 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.269209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"]
Sep 15 19:12:07.331250 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.331215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2488s\" (UniqueName: \"kubernetes.io/projected/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-kube-api-access-2488s\") pod \"payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv\" (UID: \"3969adce-1f2b-441b-b6d0-3beed6ee9fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:07.331403 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.331269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv\" (UID: \"3969adce-1f2b-441b-b6d0-3beed6ee9fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:07.331403 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.331335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxksn\" (UniqueName: \"kubernetes.io/projected/10a5f2d4-0657-4719-988c-2ed51142e686-kube-api-access-bxksn\") pod \"payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6\" (UID: \"10a5f2d4-0657-4719-988c-2ed51142e686\") " pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:07.331492 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.331444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a5f2d4-0657-4719-988c-2ed51142e686-plugins-config-volume\") pod \"payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6\" (UID: \"10a5f2d4-0657-4719-988c-2ed51142e686\") " pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:07.431975 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.431938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a5f2d4-0657-4719-988c-2ed51142e686-plugins-config-volume\") pod \"payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6\" (UID: \"10a5f2d4-0657-4719-988c-2ed51142e686\") " pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:07.432138 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.432003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2488s\" (UniqueName: \"kubernetes.io/projected/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-kube-api-access-2488s\") pod \"payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv\" (UID: \"3969adce-1f2b-441b-b6d0-3beed6ee9fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:07.432138 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.432030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv\" (UID: \"3969adce-1f2b-441b-b6d0-3beed6ee9fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:07.432138 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.432046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxksn\" (UniqueName: \"kubernetes.io/projected/10a5f2d4-0657-4719-988c-2ed51142e686-kube-api-access-bxksn\") pod \"payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6\" (UID: \"10a5f2d4-0657-4719-988c-2ed51142e686\") " pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:07.432713 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.432688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a5f2d4-0657-4719-988c-2ed51142e686-plugins-config-volume\") pod \"payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6\" (UID: \"10a5f2d4-0657-4719-988c-2ed51142e686\") " pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:07.432758 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.432696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv\" (UID: \"3969adce-1f2b-441b-b6d0-3beed6ee9fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:07.439516 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.439493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2488s\" (UniqueName: \"kubernetes.io/projected/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-kube-api-access-2488s\") pod \"payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv\" (UID: \"3969adce-1f2b-441b-b6d0-3beed6ee9fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:07.440201 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.440177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxksn\" (UniqueName: \"kubernetes.io/projected/10a5f2d4-0657-4719-988c-2ed51142e686-kube-api-access-bxksn\") pod \"payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6\" (UID: \"10a5f2d4-0657-4719-988c-2ed51142e686\") " pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:07.536291 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.536229    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:07.568301 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.568273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:07.802723 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.802699    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"]
Sep 15 19:12:07.806790 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:12:07.806764    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3969adce_1f2b_441b_b6d0_3beed6ee9fbf.slice/crio-4feb49748f32167c3cac9a5a0bc85ef6458bd914804e93222f2c94d9ff42cc7e WatchSource:0}: Error finding container 4feb49748f32167c3cac9a5a0bc85ef6458bd914804e93222f2c94d9ff42cc7e: Status 404 returned error can't find the container with id 4feb49748f32167c3cac9a5a0bc85ef6458bd914804e93222f2c94d9ff42cc7e
Sep 15 19:12:07.841558 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:07.841535    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"]
Sep 15 19:12:07.850502 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:12:07.850477    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a5f2d4_0657_4719_988c_2ed51142e686.slice/crio-2062f2702553d4f0a7c0835a6161ef16a3e43a0397fba8ae0237b4b6bda21d5d WatchSource:0}: Error finding container 2062f2702553d4f0a7c0835a6161ef16a3e43a0397fba8ae0237b4b6bda21d5d: Status 404 returned error can't find the container with id 2062f2702553d4f0a7c0835a6161ef16a3e43a0397fba8ae0237b4b6bda21d5d
Sep 15 19:12:08.092287 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.092203    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6" event={"ID":"10a5f2d4-0657-4719-988c-2ed51142e686","Type":"ContainerStarted","Data":"7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08"}
Sep 15 19:12:08.092287 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.092253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6" event={"ID":"10a5f2d4-0657-4719-988c-2ed51142e686","Type":"ContainerStarted","Data":"2062f2702553d4f0a7c0835a6161ef16a3e43a0397fba8ae0237b4b6bda21d5d"}
Sep 15 19:12:08.092287 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.092277    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:08.099746 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.099721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv" event={"ID":"3969adce-1f2b-441b-b6d0-3beed6ee9fbf","Type":"ContainerStarted","Data":"35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484"}
Sep 15 19:12:08.099891 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.099768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:08.099891 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.099788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv" event={"ID":"3969adce-1f2b-441b-b6d0-3beed6ee9fbf","Type":"ContainerStarted","Data":"4feb49748f32167c3cac9a5a0bc85ef6458bd914804e93222f2c94d9ff42cc7e"}
Sep 15 19:12:08.099891 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.099832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf" event={"ID":"6a49f0b2-bb8e-4265-8dbf-e16732793637","Type":"ContainerStarted","Data":"e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9"}
Sep 15 19:12:08.099891 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.099860    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:08.099891 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.099873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf" event={"ID":"6a49f0b2-bb8e-4265-8dbf-e16732793637","Type":"ContainerStarted","Data":"907fde54236c29e5cd1c7c6c91d446b999a4be8fe1f1b55c0574259713df9860"}
Sep 15 19:12:08.108610 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.108568    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6" podStartSLOduration=1.108555097 podStartE2EDuration="1.108555097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:12:08.106477502 +0000 UTC m=+1278.648573026" watchObservedRunningTime="2026-09-15 19:12:08.108555097 +0000 UTC m=+1278.650650625"
Sep 15 19:12:08.127486 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.127436    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf" podStartSLOduration=1.5597840509999998 podStartE2EDuration="2.127419307s" totalImagesPullingTime="567.635256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:12:08.126964768 +0000 UTC m=+1278.669060303" watchObservedRunningTime="2026-09-15 19:12:08.127419307 +0000 UTC m=+1278.669514838"
Sep 15 19:12:08.142188 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:08.142140    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv" podStartSLOduration=1.142123063 podStartE2EDuration="1.142123063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:12:08.141084448 +0000 UTC m=+1278.683179979" watchObservedRunningTime="2026-09-15 19:12:08.142123063 +0000 UTC m=+1278.684218593"
Sep 15 19:12:14.106698 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:14.106670    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:19.103519 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:19.103487    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:19.103940 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:19.103638    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:34.975279 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:34.975238    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:12:46.358030 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.357992    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"]
Sep 15 19:12:46.358571 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.358234    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf" podUID="6a49f0b2-bb8e-4265-8dbf-e16732793637" containerName="maas-api" containerID="cri-o://e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9" gracePeriod=30
Sep 15 19:12:46.408096 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.408061    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"]
Sep 15 19:12:46.408397 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.408338    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6" podUID="10a5f2d4-0657-4719-988c-2ed51142e686" containerName="payload-processing" containerID="cri-o://7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08" gracePeriod=30
Sep 15 19:12:46.413912 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.413885    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"]
Sep 15 19:12:46.414129 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.414100    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv" podUID="3969adce-1f2b-441b-b6d0-3beed6ee9fbf" containerName="payload-pre-processing" containerID="cri-o://35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484" gracePeriod=30
Sep 15 19:12:46.677236 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.677195    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:46.680449 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.680425    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:46.688583 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.688562    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:46.756848 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.756820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkp6m\" (UniqueName: \"kubernetes.io/projected/6a49f0b2-bb8e-4265-8dbf-e16732793637-kube-api-access-qkp6m\") pod \"6a49f0b2-bb8e-4265-8dbf-e16732793637\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") "
Sep 15 19:12:46.756848 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.756853    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a49f0b2-bb8e-4265-8dbf-e16732793637-openshift-service-ca\") pod \"6a49f0b2-bb8e-4265-8dbf-e16732793637\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") "
Sep 15 19:12:46.757082 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.756899    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-maas-api-tls\") pod \"6a49f0b2-bb8e-4265-8dbf-e16732793637\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") "
Sep 15 19:12:46.757082 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.756915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2488s\" (UniqueName: \"kubernetes.io/projected/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-kube-api-access-2488s\") pod \"3969adce-1f2b-441b-b6d0-3beed6ee9fbf\" (UID: \"3969adce-1f2b-441b-b6d0-3beed6ee9fbf\") "
Sep 15 19:12:46.757082 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.756948    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxksn\" (UniqueName: \"kubernetes.io/projected/10a5f2d4-0657-4719-988c-2ed51142e686-kube-api-access-bxksn\") pod \"10a5f2d4-0657-4719-988c-2ed51142e686\" (UID: \"10a5f2d4-0657-4719-988c-2ed51142e686\") "
Sep 15 19:12:46.757082 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.756975    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a5f2d4-0657-4719-988c-2ed51142e686-plugins-config-volume\") pod \"10a5f2d4-0657-4719-988c-2ed51142e686\" (UID: \"10a5f2d4-0657-4719-988c-2ed51142e686\") "
Sep 15 19:12:46.757082 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.757015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-metrics-certs\") pod \"6a49f0b2-bb8e-4265-8dbf-e16732793637\" (UID: \"6a49f0b2-bb8e-4265-8dbf-e16732793637\") "
Sep 15 19:12:46.757082 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.757042    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-plugins-config-volume\") pod \"3969adce-1f2b-441b-b6d0-3beed6ee9fbf\" (UID: \"3969adce-1f2b-441b-b6d0-3beed6ee9fbf\") "
Sep 15 19:12:46.757406 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.757374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a49f0b2-bb8e-4265-8dbf-e16732793637-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a49f0b2-bb8e-4265-8dbf-e16732793637" (UID: "6a49f0b2-bb8e-4265-8dbf-e16732793637"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:12:46.757609 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.757573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a5f2d4-0657-4719-988c-2ed51142e686-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10a5f2d4-0657-4719-988c-2ed51142e686" (UID: "10a5f2d4-0657-4719-988c-2ed51142e686"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:12:46.757609 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.757601    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3969adce-1f2b-441b-b6d0-3beed6ee9fbf" (UID: "3969adce-1f2b-441b-b6d0-3beed6ee9fbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:12:46.759628 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.759604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-kube-api-access-2488s" (OuterVolumeSpecName: "kube-api-access-2488s") pod "3969adce-1f2b-441b-b6d0-3beed6ee9fbf" (UID: "3969adce-1f2b-441b-b6d0-3beed6ee9fbf"). InnerVolumeSpecName "kube-api-access-2488s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:12:46.759740 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.759715    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6a49f0b2-bb8e-4265-8dbf-e16732793637" (UID: "6a49f0b2-bb8e-4265-8dbf-e16732793637"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:12:46.759864 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.759724    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a5f2d4-0657-4719-988c-2ed51142e686-kube-api-access-bxksn" (OuterVolumeSpecName: "kube-api-access-bxksn") pod "10a5f2d4-0657-4719-988c-2ed51142e686" (UID: "10a5f2d4-0657-4719-988c-2ed51142e686"). InnerVolumeSpecName "kube-api-access-bxksn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:12:46.759864 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.759764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a49f0b2-bb8e-4265-8dbf-e16732793637-kube-api-access-qkp6m" (OuterVolumeSpecName: "kube-api-access-qkp6m") pod "6a49f0b2-bb8e-4265-8dbf-e16732793637" (UID: "6a49f0b2-bb8e-4265-8dbf-e16732793637"). InnerVolumeSpecName "kube-api-access-qkp6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:12:46.759864 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.759793    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a49f0b2-bb8e-4265-8dbf-e16732793637" (UID: "6a49f0b2-bb8e-4265-8dbf-e16732793637"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:12:46.857880 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.857848    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkp6m\" (UniqueName: \"kubernetes.io/projected/6a49f0b2-bb8e-4265-8dbf-e16732793637-kube-api-access-qkp6m\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:12:46.857880 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.857875    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a49f0b2-bb8e-4265-8dbf-e16732793637-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:12:46.857880 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.857885    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:12:46.858084 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.857895    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2488s\" (UniqueName: \"kubernetes.io/projected/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-kube-api-access-2488s\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:12:46.858084 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.857904    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxksn\" (UniqueName: \"kubernetes.io/projected/10a5f2d4-0657-4719-988c-2ed51142e686-kube-api-access-bxksn\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:12:46.858084 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.857912    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a5f2d4-0657-4719-988c-2ed51142e686-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:12:46.858084 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.857920    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a49f0b2-bb8e-4265-8dbf-e16732793637-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:12:46.858084 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:46.857928    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3969adce-1f2b-441b-b6d0-3beed6ee9fbf-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:12:47.272782 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.272743    2578 generic.go:356] "Generic (PLEG): container finished" podID="10a5f2d4-0657-4719-988c-2ed51142e686" containerID="7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08" exitCode=0
Sep 15 19:12:47.272959 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.272792    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"
Sep 15 19:12:47.272959 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.272836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6" event={"ID":"10a5f2d4-0657-4719-988c-2ed51142e686","Type":"ContainerDied","Data":"7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08"}
Sep 15 19:12:47.272959 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.272888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6" event={"ID":"10a5f2d4-0657-4719-988c-2ed51142e686","Type":"ContainerDied","Data":"2062f2702553d4f0a7c0835a6161ef16a3e43a0397fba8ae0237b4b6bda21d5d"}
Sep 15 19:12:47.272959 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.272909    2578 scope.go:117] "RemoveContainer" containerID="7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08"
Sep 15 19:12:47.274164 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.274141    2578 generic.go:356] "Generic (PLEG): container finished" podID="3969adce-1f2b-441b-b6d0-3beed6ee9fbf" containerID="35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484" exitCode=0
Sep 15 19:12:47.274283 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.274166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv" event={"ID":"3969adce-1f2b-441b-b6d0-3beed6ee9fbf","Type":"ContainerDied","Data":"35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484"}
Sep 15 19:12:47.274283 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.274185    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"
Sep 15 19:12:47.274283 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.274216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv" event={"ID":"3969adce-1f2b-441b-b6d0-3beed6ee9fbf","Type":"ContainerDied","Data":"4feb49748f32167c3cac9a5a0bc85ef6458bd914804e93222f2c94d9ff42cc7e"}
Sep 15 19:12:47.275541 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.275519    2578 generic.go:356] "Generic (PLEG): container finished" podID="6a49f0b2-bb8e-4265-8dbf-e16732793637" containerID="e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9" exitCode=0
Sep 15 19:12:47.275691 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.275566    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"
Sep 15 19:12:47.275691 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.275577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf" event={"ID":"6a49f0b2-bb8e-4265-8dbf-e16732793637","Type":"ContainerDied","Data":"e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9"}
Sep 15 19:12:47.275691 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.275611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf" event={"ID":"6a49f0b2-bb8e-4265-8dbf-e16732793637","Type":"ContainerDied","Data":"907fde54236c29e5cd1c7c6c91d446b999a4be8fe1f1b55c0574259713df9860"}
Sep 15 19:12:47.285875 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.285784    2578 scope.go:117] "RemoveContainer" containerID="7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08"
Sep 15 19:12:47.286242 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:12:47.286218    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08\": container with ID starting with 7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08 not found: ID does not exist" containerID="7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08"
Sep 15 19:12:47.286377 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.286355    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08"} err="failed to get container status \"7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08\": rpc error: code = NotFound desc = could not find container \"7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08\": container with ID starting with 7a2ffd2788857717a83aa0106f7475e8bec87a5ab149f5138952c4ca3ba41d08 not found: ID does not exist"
Sep 15 19:12:47.286469 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.286457    2578 scope.go:117] "RemoveContainer" containerID="35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484"
Sep 15 19:12:47.296454 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.296430    2578 scope.go:117] "RemoveContainer" containerID="35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484"
Sep 15 19:12:47.296693 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:12:47.296673    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484\": container with ID starting with 35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484 not found: ID does not exist" containerID="35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484"
Sep 15 19:12:47.296744 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.296700    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484"} err="failed to get container status \"35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484\": rpc error: code = NotFound desc = could not find container \"35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484\": container with ID starting with 35c4099cf16222e26408f01ce91e5dd41925ebe889525819cf141c0ef6cd2484 not found: ID does not exist"
Sep 15 19:12:47.296744 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.296715    2578 scope.go:117] "RemoveContainer" containerID="e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9"
Sep 15 19:12:47.303191 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.303166    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"]
Sep 15 19:12:47.304554 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.304541    2578 scope.go:117] "RemoveContainer" containerID="e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9"
Sep 15 19:12:47.304808 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:12:47.304780    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9\": container with ID starting with e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9 not found: ID does not exist" containerID="e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9"
Sep 15 19:12:47.304869 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.304821    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9"} err="failed to get container status \"e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9\": rpc error: code = NotFound desc = could not find container \"e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9\": container with ID starting with e83d78abb84ef6e9ce6bc6cad1aa479ca4d82b90493088433da106580d7a55c9 not found: ID does not exist"
Sep 15 19:12:47.309048 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.309027    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a0497c3b-5fb5d5f5c6-krcqf"]
Sep 15 19:12:47.318232 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.318212    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"]
Sep 15 19:12:47.323815 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.323775    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a0497c3b-9d7897c49-2fdz6"]
Sep 15 19:12:47.333854 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.333832    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"]
Sep 15 19:12:47.337607 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:47.337587    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a0497c3b-7f989864f8-8gchv"]
Sep 15 19:12:48.098278 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:48.098244    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a5f2d4-0657-4719-988c-2ed51142e686" path="/var/lib/kubelet/pods/10a5f2d4-0657-4719-988c-2ed51142e686/volumes"
Sep 15 19:12:48.098660 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:48.098596    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3969adce-1f2b-441b-b6d0-3beed6ee9fbf" path="/var/lib/kubelet/pods/3969adce-1f2b-441b-b6d0-3beed6ee9fbf/volumes"
Sep 15 19:12:48.098953 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:48.098939    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a49f0b2-bb8e-4265-8dbf-e16732793637" path="/var/lib/kubelet/pods/6a49f0b2-bb8e-4265-8dbf-e16732793637/volumes"
Sep 15 19:12:59.278613 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:12:59.278573    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:13:06.481932 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.481896    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"]
Sep 15 19:13:06.482314 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.482226    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a49f0b2-bb8e-4265-8dbf-e16732793637" containerName="maas-api"
Sep 15 19:13:06.482314 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.482253    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a49f0b2-bb8e-4265-8dbf-e16732793637" containerName="maas-api"
Sep 15 19:13:06.482314 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.482268    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3969adce-1f2b-441b-b6d0-3beed6ee9fbf" containerName="payload-pre-processing"
Sep 15 19:13:06.482314 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.482275    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3969adce-1f2b-441b-b6d0-3beed6ee9fbf" containerName="payload-pre-processing"
Sep 15 19:13:06.482314 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.482295    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a5f2d4-0657-4719-988c-2ed51142e686" containerName="payload-processing"
Sep 15 19:13:06.482314 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.482303    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a5f2d4-0657-4719-988c-2ed51142e686" containerName="payload-processing"
Sep 15 19:13:06.482606 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.482399    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3969adce-1f2b-441b-b6d0-3beed6ee9fbf" containerName="payload-pre-processing"
Sep 15 19:13:06.482606 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.482412    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a5f2d4-0657-4719-988c-2ed51142e686" containerName="payload-processing"
Sep 15 19:13:06.482606 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.482421    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a49f0b2-bb8e-4265-8dbf-e16732793637" containerName="maas-api"
Sep 15 19:13:06.486741 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.486721    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.490087 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.490063    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"kube-root-ca.crt\""
Sep 15 19:13:06.490204 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.490063    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"distinct-w0-900c88-kserve-self-signed-certs\""
Sep 15 19:13:06.490204 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.490102    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"openshift-service-ca.crt\""
Sep 15 19:13:06.490384 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.490123    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"default-dockercfg-6plq7\""
Sep 15 19:13:06.493164 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.493097    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"]
Sep 15 19:13:06.623875 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.623839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-dshm\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.624040 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.623902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tls-certs\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.624040 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.623988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-home\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.624040 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.624020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tmp-dir\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.624142 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.624043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-model-cache\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.624142 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.624104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzlrg\" (UniqueName: \"kubernetes.io/projected/33fda78e-3d3b-4288-8368-f7e7d0c518f2-kube-api-access-tzlrg\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.725147 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.725108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-home\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.725147 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.725150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tmp-dir\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.725403 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.725168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-model-cache\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.725403 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.725279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzlrg\" (UniqueName: \"kubernetes.io/projected/33fda78e-3d3b-4288-8368-f7e7d0c518f2-kube-api-access-tzlrg\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.725403 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.725336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-dshm\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.725403 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.725393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tls-certs\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.725606 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.725552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tmp-dir\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.725663 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.725622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-home\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.725697 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.725652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-model-cache\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.727534 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.727515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-dshm\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.727971 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.727952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tls-certs\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.733791 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.733744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzlrg\" (UniqueName: \"kubernetes.io/projected/33fda78e-3d3b-4288-8368-f7e7d0c518f2-kube-api-access-tzlrg\") pod \"distinct-w0-900c88-kserve-c858876c9-x9pf6\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") " pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:06.797640 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:06.797605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:07.073198 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:07.073168    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"]
Sep 15 19:13:07.075597 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:13:07.075570    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33fda78e_3d3b_4288_8368_f7e7d0c518f2.slice/crio-ba8a157e709b7d64cf26f384145188a5189c63cc259191785f304706b537d941 WatchSource:0}: Error finding container ba8a157e709b7d64cf26f384145188a5189c63cc259191785f304706b537d941: Status 404 returned error can't find the container with id ba8a157e709b7d64cf26f384145188a5189c63cc259191785f304706b537d941
Sep 15 19:13:07.366605 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:07.366518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6" event={"ID":"33fda78e-3d3b-4288-8368-f7e7d0c518f2","Type":"ContainerStarted","Data":"10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a"}
Sep 15 19:13:07.366605 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:07.366560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6" event={"ID":"33fda78e-3d3b-4288-8368-f7e7d0c518f2","Type":"ContainerStarted","Data":"ba8a157e709b7d64cf26f384145188a5189c63cc259191785f304706b537d941"}
Sep 15 19:13:07.382627 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:07.382569    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6" podStartSLOduration=1.382553635 podStartE2EDuration="1.382553635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:13:07.382171876 +0000 UTC m=+1337.924267406" watchObservedRunningTime="2026-09-15 19:13:07.382553635 +0000 UTC m=+1337.924649164"
Sep 15 19:13:12.395324 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.395279    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"]
Sep 15 19:13:12.399129 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.399107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.401275 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.401252    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-362ea4-serving-cert\""
Sep 15 19:13:12.409571 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.409439    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"]
Sep 15 19:13:12.469873 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.469842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.470050 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.469890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.470050 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.469965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44qk\" (UniqueName: \"kubernetes.io/projected/e55df82e-cab7-4254-8331-0280be195c4d-kube-api-access-h44qk\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.470050 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.470043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55df82e-cab7-4254-8331-0280be195c4d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.571179 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.571126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.571179 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.571187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.571412 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.571220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44qk\" (UniqueName: \"kubernetes.io/projected/e55df82e-cab7-4254-8331-0280be195c4d-kube-api-access-h44qk\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.571412 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.571276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55df82e-cab7-4254-8331-0280be195c4d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.571412 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:13:12.571349    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-serving-cert: secret "maas-api-e2e-shared-a-w1-362ea4-serving-cert" not found
Sep 15 19:13:12.571517 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:13:12.571429    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-maas-api-tls podName:e55df82e-cab7-4254-8331-0280be195c4d nodeName:}" failed. No retries permitted until 2026-09-15 19:13:13.071412533 +0000 UTC m=+1343.613508057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-maas-api-tls") pod "maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9" (UID: "e55df82e-cab7-4254-8331-0280be195c4d") : secret "maas-api-e2e-shared-a-w1-362ea4-serving-cert" not found
Sep 15 19:13:12.571954 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.571935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55df82e-cab7-4254-8331-0280be195c4d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.573595 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.573575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:12.579087 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:12.579063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44qk\" (UniqueName: \"kubernetes.io/projected/e55df82e-cab7-4254-8331-0280be195c4d-kube-api-access-h44qk\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:13.075858 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.075817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:13.078234 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.078213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:13.134049 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.134013    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"]
Sep 15 19:13:13.137478 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.137462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:13:13.139751 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.139727    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-362ea4-dockercfg-bwmcf\""
Sep 15 19:13:13.139751 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.139740    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-362ea4\""
Sep 15 19:13:13.147972 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.147947    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"]
Sep 15 19:13:13.158113 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.158080    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"]
Sep 15 19:13:13.161395 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.161373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:13:13.173684 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.173658    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"]
Sep 15 19:13:13.277062 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.277025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86sv8\" (UniqueName: \"kubernetes.io/projected/1f1f00a0-4794-4481-9d71-fb795ae77e91-kube-api-access-86sv8\") pod \"payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4\" (UID: \"1f1f00a0-4794-4481-9d71-fb795ae77e91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:13:13.277260 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.277073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfsb\" (UniqueName: \"kubernetes.io/projected/493dfe32-ca79-4d34-9775-116c767bc1d1-kube-api-access-4sfsb\") pod \"payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s\" (UID: \"493dfe32-ca79-4d34-9775-116c767bc1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:13:13.277260 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.277146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1f00a0-4794-4481-9d71-fb795ae77e91-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4\" (UID: \"1f1f00a0-4794-4481-9d71-fb795ae77e91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:13:13.277260 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.277178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493dfe32-ca79-4d34-9775-116c767bc1d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s\" (UID: \"493dfe32-ca79-4d34-9775-116c767bc1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:13:13.311889 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.311855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:13.378501 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.378115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86sv8\" (UniqueName: \"kubernetes.io/projected/1f1f00a0-4794-4481-9d71-fb795ae77e91-kube-api-access-86sv8\") pod \"payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4\" (UID: \"1f1f00a0-4794-4481-9d71-fb795ae77e91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:13:13.378501 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.378175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfsb\" (UniqueName: \"kubernetes.io/projected/493dfe32-ca79-4d34-9775-116c767bc1d1-kube-api-access-4sfsb\") pod \"payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s\" (UID: \"493dfe32-ca79-4d34-9775-116c767bc1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:13:13.378501 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.378240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1f00a0-4794-4481-9d71-fb795ae77e91-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4\" (UID: \"1f1f00a0-4794-4481-9d71-fb795ae77e91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:13:13.378501 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.378266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493dfe32-ca79-4d34-9775-116c767bc1d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s\" (UID: \"493dfe32-ca79-4d34-9775-116c767bc1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:13:13.379130 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.379107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493dfe32-ca79-4d34-9775-116c767bc1d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s\" (UID: \"493dfe32-ca79-4d34-9775-116c767bc1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:13:13.379504 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.379477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1f00a0-4794-4481-9d71-fb795ae77e91-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4\" (UID: \"1f1f00a0-4794-4481-9d71-fb795ae77e91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:13:13.386593 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.386573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfsb\" (UniqueName: \"kubernetes.io/projected/493dfe32-ca79-4d34-9775-116c767bc1d1-kube-api-access-4sfsb\") pod \"payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s\" (UID: \"493dfe32-ca79-4d34-9775-116c767bc1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:13:13.386692 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.386671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86sv8\" (UniqueName: \"kubernetes.io/projected/1f1f00a0-4794-4481-9d71-fb795ae77e91-kube-api-access-86sv8\") pod \"payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4\" (UID: \"1f1f00a0-4794-4481-9d71-fb795ae77e91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:13:13.449067 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.449025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:13:13.471848 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.471821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:13:13.584425 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.584398    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"]
Sep 15 19:13:13.586553 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:13:13.586529    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55df82e_cab7_4254_8331_0280be195c4d.slice/crio-b16ba718ba1b1535c8aa2d0a1bb8869acd002bec203e7cd22533e294eae73ebe WatchSource:0}: Error finding container b16ba718ba1b1535c8aa2d0a1bb8869acd002bec203e7cd22533e294eae73ebe: Status 404 returned error can't find the container with id b16ba718ba1b1535c8aa2d0a1bb8869acd002bec203e7cd22533e294eae73ebe
Sep 15 19:13:13.587960 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.587943    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:13:13.724991 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.724958    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"]
Sep 15 19:13:13.728579 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:13:13.728556    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493dfe32_ca79_4d34_9775_116c767bc1d1.slice/crio-a3cfbb476f5f8ed3bd576c3a1e6c497531788c24aff88439afb5a037d9f6ea12 WatchSource:0}: Error finding container a3cfbb476f5f8ed3bd576c3a1e6c497531788c24aff88439afb5a037d9f6ea12: Status 404 returned error can't find the container with id a3cfbb476f5f8ed3bd576c3a1e6c497531788c24aff88439afb5a037d9f6ea12
Sep 15 19:13:13.752482 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:13.752446    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"]
Sep 15 19:13:13.753987 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:13:13.753964    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1f00a0_4794_4481_9d71_fb795ae77e91.slice/crio-2ff33c8cf59a7279e439917823b557ebc9380322c3c89f32436aa97a566b7a2e WatchSource:0}: Error finding container 2ff33c8cf59a7279e439917823b557ebc9380322c3c89f32436aa97a566b7a2e: Status 404 returned error can't find the container with id 2ff33c8cf59a7279e439917823b557ebc9380322c3c89f32436aa97a566b7a2e
Sep 15 19:13:14.402092 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.402051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s" event={"ID":"493dfe32-ca79-4d34-9775-116c767bc1d1","Type":"ContainerStarted","Data":"a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1"}
Sep 15 19:13:14.402297 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.402116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s" event={"ID":"493dfe32-ca79-4d34-9775-116c767bc1d1","Type":"ContainerStarted","Data":"a3cfbb476f5f8ed3bd576c3a1e6c497531788c24aff88439afb5a037d9f6ea12"}
Sep 15 19:13:14.402297 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.402156    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:13:14.403820 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.403780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9" event={"ID":"e55df82e-cab7-4254-8331-0280be195c4d","Type":"ContainerStarted","Data":"1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f"}
Sep 15 19:13:14.403955 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.403835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9" event={"ID":"e55df82e-cab7-4254-8331-0280be195c4d","Type":"ContainerStarted","Data":"b16ba718ba1b1535c8aa2d0a1bb8869acd002bec203e7cd22533e294eae73ebe"}
Sep 15 19:13:14.403955 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.403881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:14.405362 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.405339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4" event={"ID":"1f1f00a0-4794-4481-9d71-fb795ae77e91","Type":"ContainerStarted","Data":"71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61"}
Sep 15 19:13:14.405468 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.405383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4" event={"ID":"1f1f00a0-4794-4481-9d71-fb795ae77e91","Type":"ContainerStarted","Data":"2ff33c8cf59a7279e439917823b557ebc9380322c3c89f32436aa97a566b7a2e"}
Sep 15 19:13:14.405468 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.405422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:13:14.419020 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.418967    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s" podStartSLOduration=1.418949047 podStartE2EDuration="1.418949047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:13:14.417048842 +0000 UTC m=+1344.959144370" watchObservedRunningTime="2026-09-15 19:13:14.418949047 +0000 UTC m=+1344.961044582"
Sep 15 19:13:14.434747 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.434700    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9" podStartSLOduration=2.016096258 podStartE2EDuration="2.434684723s" totalImagesPullingTime="418.588465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:13:14.433543315 +0000 UTC m=+1344.975638845" watchObservedRunningTime="2026-09-15 19:13:14.434684723 +0000 UTC m=+1344.976780252"
Sep 15 19:13:14.451598 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:14.451547    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4" podStartSLOduration=1.451527035 podStartE2EDuration="1.451527035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:13:14.449138482 +0000 UTC m=+1344.991234012" watchObservedRunningTime="2026-09-15 19:13:14.451527035 +0000 UTC m=+1344.993622566"
Sep 15 19:13:16.798389 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:16.798352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:16.798389 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:16.798399    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:16.810893 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:16.810868    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:17.430543 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:17.430513    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:13:20.414981 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:20.414950    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:13:25.411920 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:25.411888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:13:25.412315 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:25.411963    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:13:30.178252 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:30.178217    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:13:39.784564 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.784525    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"]
Sep 15 19:13:39.790875 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.789470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:13:39.792340 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.792310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-1b0e3f\""
Sep 15 19:13:39.792547 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.792315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-1b0e3f-dockercfg-hj6h2\""
Sep 15 19:13:39.796276 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.796252    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"]
Sep 15 19:13:39.804550 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.804530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7992\" (UniqueName: \"kubernetes.io/projected/50f9e362-7091-4a00-bb76-f651a1588bc7-kube-api-access-k7992\") pod \"payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp\" (UID: \"50f9e362-7091-4a00-bb76-f651a1588bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:13:39.804649 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.804606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f9e362-7091-4a00-bb76-f651a1588bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp\" (UID: \"50f9e362-7091-4a00-bb76-f651a1588bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:13:39.909826 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.909775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f9e362-7091-4a00-bb76-f651a1588bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp\" (UID: \"50f9e362-7091-4a00-bb76-f651a1588bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:13:39.910010 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.909844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7992\" (UniqueName: \"kubernetes.io/projected/50f9e362-7091-4a00-bb76-f651a1588bc7-kube-api-access-k7992\") pod \"payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp\" (UID: \"50f9e362-7091-4a00-bb76-f651a1588bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:13:39.910499 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.910478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f9e362-7091-4a00-bb76-f651a1588bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp\" (UID: \"50f9e362-7091-4a00-bb76-f651a1588bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:13:39.917711 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:39.917684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7992\" (UniqueName: \"kubernetes.io/projected/50f9e362-7091-4a00-bb76-f651a1588bc7-kube-api-access-k7992\") pod \"payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp\" (UID: \"50f9e362-7091-4a00-bb76-f651a1588bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:13:40.102560 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:40.102465    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:13:40.373684 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:40.373657    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"]
Sep 15 19:13:40.376192 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:13:40.376160    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f9e362_7091_4a00_bb76_f651a1588bc7.slice/crio-e2d6982c4e9af97d34c3beba70dc967c71220116d6e43598f83261f33a74eb27 WatchSource:0}: Error finding container e2d6982c4e9af97d34c3beba70dc967c71220116d6e43598f83261f33a74eb27: Status 404 returned error can't find the container with id e2d6982c4e9af97d34c3beba70dc967c71220116d6e43598f83261f33a74eb27
Sep 15 19:13:40.526267 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:40.526230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp" event={"ID":"50f9e362-7091-4a00-bb76-f651a1588bc7","Type":"ContainerStarted","Data":"39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7"}
Sep 15 19:13:40.526432 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:40.526293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp" event={"ID":"50f9e362-7091-4a00-bb76-f651a1588bc7","Type":"ContainerStarted","Data":"e2d6982c4e9af97d34c3beba70dc967c71220116d6e43598f83261f33a74eb27"}
Sep 15 19:13:40.526432 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:40.526373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:13:40.541193 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:40.541139    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp" podStartSLOduration=1.541126268 podStartE2EDuration="1.541126268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:13:40.539002775 +0000 UTC m=+1371.081098303" watchObservedRunningTime="2026-09-15 19:13:40.541126268 +0000 UTC m=+1371.083221797"
Sep 15 19:13:51.533750 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:13:51.533718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:14:03.003392 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:03.003300    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8999-07396e"]
Sep 15 19:14:03.006306 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:03.006283    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-07396e"
Sep 15 19:14:03.008809 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:03.008777    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgkcn\""
Sep 15 19:14:03.014409 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:03.014385    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8999-07396e"]
Sep 15 19:14:03.107391 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:03.107344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg8lv\" (UniqueName: \"kubernetes.io/projected/f095ec3f-46ec-4b9a-a88e-fd7f1c183d05-kube-api-access-cg8lv\") pod \"test-curl-8999-07396e\" (UID: \"f095ec3f-46ec-4b9a-a88e-fd7f1c183d05\") " pod="opendatahub/test-curl-8999-07396e"
Sep 15 19:14:03.208486 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:03.208445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg8lv\" (UniqueName: \"kubernetes.io/projected/f095ec3f-46ec-4b9a-a88e-fd7f1c183d05-kube-api-access-cg8lv\") pod \"test-curl-8999-07396e\" (UID: \"f095ec3f-46ec-4b9a-a88e-fd7f1c183d05\") " pod="opendatahub/test-curl-8999-07396e"
Sep 15 19:14:03.217439 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:03.217408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg8lv\" (UniqueName: \"kubernetes.io/projected/f095ec3f-46ec-4b9a-a88e-fd7f1c183d05-kube-api-access-cg8lv\") pod \"test-curl-8999-07396e\" (UID: \"f095ec3f-46ec-4b9a-a88e-fd7f1c183d05\") " pod="opendatahub/test-curl-8999-07396e"
Sep 15 19:14:03.317113 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:03.316716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-07396e"
Sep 15 19:14:03.587985 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:03.587960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8999-07396e"]
Sep 15 19:14:03.590393 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:14:03.590363    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf095ec3f_46ec_4b9a_a88e_fd7f1c183d05.slice/crio-dcb43ee0cd7c38c620a712bc12e0a456de8fcdbac7e1341fa0f05a5873c58f7e WatchSource:0}: Error finding container dcb43ee0cd7c38c620a712bc12e0a456de8fcdbac7e1341fa0f05a5873c58f7e: Status 404 returned error can't find the container with id dcb43ee0cd7c38c620a712bc12e0a456de8fcdbac7e1341fa0f05a5873c58f7e
Sep 15 19:14:03.643054 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:03.643027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8999-07396e" event={"ID":"f095ec3f-46ec-4b9a-a88e-fd7f1c183d05","Type":"ContainerStarted","Data":"dcb43ee0cd7c38c620a712bc12e0a456de8fcdbac7e1341fa0f05a5873c58f7e"}
Sep 15 19:14:04.649204 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:04.649163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8999-07396e" event={"ID":"f095ec3f-46ec-4b9a-a88e-fd7f1c183d05","Type":"ContainerStarted","Data":"a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145"}
Sep 15 19:14:04.680021 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:04.679967    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8999-07396e" podStartSLOduration=2.679952559 podStartE2EDuration="2.679952559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:14:04.678777805 +0000 UTC m=+1395.220873335" watchObservedRunningTime="2026-09-15 19:14:04.679952559 +0000 UTC m=+1395.222048090"
Sep 15 19:14:05.240494 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.240461    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8999-07396e"]
Sep 15 19:14:05.243769 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.243742    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8999-07396e"]
Sep 15 19:14:05.353122 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.353061    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8999-672542"]
Sep 15 19:14:05.353405 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.353393    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f095ec3f-46ec-4b9a-a88e-fd7f1c183d05" containerName="test-curl-8999-07396e"
Sep 15 19:14:05.353448 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.353407    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f095ec3f-46ec-4b9a-a88e-fd7f1c183d05" containerName="test-curl-8999-07396e"
Sep 15 19:14:05.353487 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.353476    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f095ec3f-46ec-4b9a-a88e-fd7f1c183d05" containerName="test-curl-8999-07396e"
Sep 15 19:14:05.360219 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.360191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-672542"
Sep 15 19:14:05.363781 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.363759    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8999-672542"]
Sep 15 19:14:05.424720 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.424695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rqlx\" (UniqueName: \"kubernetes.io/projected/5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8-kube-api-access-2rqlx\") pod \"test-curl-8999-672542\" (UID: \"5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8\") " pod="opendatahub/test-curl-8999-672542"
Sep 15 19:14:05.525980 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.525956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rqlx\" (UniqueName: \"kubernetes.io/projected/5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8-kube-api-access-2rqlx\") pod \"test-curl-8999-672542\" (UID: \"5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8\") " pod="opendatahub/test-curl-8999-672542"
Sep 15 19:14:05.534659 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.534637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rqlx\" (UniqueName: \"kubernetes.io/projected/5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8-kube-api-access-2rqlx\") pod \"test-curl-8999-672542\" (UID: \"5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8\") " pod="opendatahub/test-curl-8999-672542"
Sep 15 19:14:05.654320 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:05.654291    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8999-07396e" container="test-curl-8999-07396e"
Sep 15 19:14:05.670318 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.670300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-672542"
Sep 15 19:14:05.937693 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:05.937571    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8999-672542"]
Sep 15 19:14:05.940104 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:14:05.940078    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d4077ab_ecfe_4dc0_a3cc_812ef9f76bc8.slice/crio-08fc4a87c76c27154add861d6a3065c4c5ea38ccbb52c62b57e576c01b775852 WatchSource:0}: Error finding container 08fc4a87c76c27154add861d6a3065c4c5ea38ccbb52c62b57e576c01b775852: Status 404 returned error can't find the container with id 08fc4a87c76c27154add861d6a3065c4c5ea38ccbb52c62b57e576c01b775852
Sep 15 19:14:06.660669 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:06.660630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8999-672542" event={"ID":"5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8","Type":"ContainerStarted","Data":"5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5"}
Sep 15 19:14:06.661126 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:06.660677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8999-672542" event={"ID":"5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8","Type":"ContainerStarted","Data":"08fc4a87c76c27154add861d6a3065c4c5ea38ccbb52c62b57e576c01b775852"}
Sep 15 19:14:06.661126 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:06.660716    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8999-07396e" podUID="f095ec3f-46ec-4b9a-a88e-fd7f1c183d05" containerName="test-curl-8999-07396e" containerID="cri-o://a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145" gracePeriod=2
Sep 15 19:14:06.674131 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:06.674090    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8999-672542" podStartSLOduration=1.674076331 podStartE2EDuration="1.674076331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:14:06.672855749 +0000 UTC m=+1397.214951279" watchObservedRunningTime="2026-09-15 19:14:06.674076331 +0000 UTC m=+1397.216171854"
Sep 15 19:14:07.229955 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.229915    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8999-672542"]
Sep 15 19:14:07.233457 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.233430    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8999-672542"]
Sep 15 19:14:07.341720 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.341646    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8999-67119e"]
Sep 15 19:14:07.342039 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.342023    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8" containerName="test-curl-8999-672542"
Sep 15 19:14:07.342084 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.342041    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8" containerName="test-curl-8999-672542"
Sep 15 19:14:07.342119 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.342112    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8" containerName="test-curl-8999-672542"
Sep 15 19:14:07.345156 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.345141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-67119e"
Sep 15 19:14:07.350792 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.350771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8999-67119e"]
Sep 15 19:14:07.442730 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.442699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9dxz\" (UniqueName: \"kubernetes.io/projected/327231f9-795b-4074-be6c-3e3408c3ff89-kube-api-access-m9dxz\") pod \"test-curl-8999-67119e\" (UID: \"327231f9-795b-4074-be6c-3e3408c3ff89\") " pod="opendatahub/test-curl-8999-67119e"
Sep 15 19:14:07.543418 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.543391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dxz\" (UniqueName: \"kubernetes.io/projected/327231f9-795b-4074-be6c-3e3408c3ff89-kube-api-access-m9dxz\") pod \"test-curl-8999-67119e\" (UID: \"327231f9-795b-4074-be6c-3e3408c3ff89\") " pod="opendatahub/test-curl-8999-67119e"
Sep 15 19:14:07.551393 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.551373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dxz\" (UniqueName: \"kubernetes.io/projected/327231f9-795b-4074-be6c-3e3408c3ff89-kube-api-access-m9dxz\") pod \"test-curl-8999-67119e\" (UID: \"327231f9-795b-4074-be6c-3e3408c3ff89\") " pod="opendatahub/test-curl-8999-67119e"
Sep 15 19:14:07.655542 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.655516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-67119e"
Sep 15 19:14:07.665325 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:07.665301    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8999-672542" container="test-curl-8999-672542"
Sep 15 19:14:07.924568 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:07.924545    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8999-67119e"]
Sep 15 19:14:07.926707 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:14:07.926679    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod327231f9_795b_4074_be6c_3e3408c3ff89.slice/crio-009b26af820c0a886ef45dc00d45aeab3caf547c0471bc6302a1e1fc901608d1 WatchSource:0}: Error finding container 009b26af820c0a886ef45dc00d45aeab3caf547c0471bc6302a1e1fc901608d1: Status 404 returned error can't find the container with id 009b26af820c0a886ef45dc00d45aeab3caf547c0471bc6302a1e1fc901608d1
Sep 15 19:14:08.672992 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:08.672959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8999-67119e" event={"ID":"327231f9-795b-4074-be6c-3e3408c3ff89","Type":"ContainerStarted","Data":"1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2"}
Sep 15 19:14:08.673358 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:08.673008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8999-67119e" event={"ID":"327231f9-795b-4074-be6c-3e3408c3ff89","Type":"ContainerStarted","Data":"009b26af820c0a886ef45dc00d45aeab3caf547c0471bc6302a1e1fc901608d1"}
Sep 15 19:14:08.673358 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:08.673091    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8999-672542" podUID="5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8" containerName="test-curl-8999-672542" containerID="cri-o://5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5" gracePeriod=2
Sep 15 19:14:08.686728 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:08.686674    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8999-67119e" podStartSLOduration=1.686660541 podStartE2EDuration="1.686660541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:14:08.685453496 +0000 UTC m=+1399.227549026" watchObservedRunningTime="2026-09-15 19:14:08.686660541 +0000 UTC m=+1399.228756069"
Sep 15 19:14:08.806860 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:08.806835    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-07396e"
Sep 15 19:14:08.956121 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:08.956058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg8lv\" (UniqueName: \"kubernetes.io/projected/f095ec3f-46ec-4b9a-a88e-fd7f1c183d05-kube-api-access-cg8lv\") pod \"f095ec3f-46ec-4b9a-a88e-fd7f1c183d05\" (UID: \"f095ec3f-46ec-4b9a-a88e-fd7f1c183d05\") "
Sep 15 19:14:08.957930 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:08.957905    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f095ec3f-46ec-4b9a-a88e-fd7f1c183d05-kube-api-access-cg8lv" (OuterVolumeSpecName: "kube-api-access-cg8lv") pod "f095ec3f-46ec-4b9a-a88e-fd7f1c183d05" (UID: "f095ec3f-46ec-4b9a-a88e-fd7f1c183d05"). InnerVolumeSpecName "kube-api-access-cg8lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:09.057603 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:09.057580    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg8lv\" (UniqueName: \"kubernetes.io/projected/f095ec3f-46ec-4b9a-a88e-fd7f1c183d05-kube-api-access-cg8lv\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:14:09.243881 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:09.243781    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8999-67119e"]
Sep 15 19:14:09.248374 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:09.248346    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8999-67119e"]
Sep 15 19:14:09.678642 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:09.678608    2578 generic.go:356] "Generic (PLEG): container finished" podID="f095ec3f-46ec-4b9a-a88e-fd7f1c183d05" containerID="a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145" exitCode=137
Sep 15 19:14:09.679067 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:09.678673    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-07396e"
Sep 15 19:14:09.679067 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:09.678709    2578 scope.go:117] "RemoveContainer" containerID="a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145"
Sep 15 19:14:09.679067 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:09.678989    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8999-67119e" container="test-curl-8999-67119e"
Sep 15 19:14:09.688455 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:09.688437    2578 scope.go:117] "RemoveContainer" containerID="a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145"
Sep 15 19:14:09.688729 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:09.688711    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145\": container with ID starting with a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145 not found: ID does not exist" containerID="a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145"
Sep 15 19:14:09.688792 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:09.688742    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145"} err="failed to get container status \"a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145\": rpc error: code = NotFound desc = could not find container \"a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145\": container with ID starting with a7e30d2c52b18290213b68424a221c898b2ff31126021d57c68998490045c145 not found: ID does not exist"
Sep 15 19:14:09.689958 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:09.689937    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8999-07396e\" is forbidden: User \"system:node:ip-10-0-135-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-221.ec2.internal' and this object" podUID="f095ec3f-46ec-4b9a-a88e-fd7f1c183d05" pod="opendatahub/test-curl-8999-07396e"
Sep 15 19:14:10.096815 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:10.096719    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8999-07396e\" is forbidden: User \"system:node:ip-10-0-135-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-221.ec2.internal' and this object" podUID="f095ec3f-46ec-4b9a-a88e-fd7f1c183d05" pod="opendatahub/test-curl-8999-07396e"
Sep 15 19:14:10.098833 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:10.098793    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f095ec3f-46ec-4b9a-a88e-fd7f1c183d05" path="/var/lib/kubelet/pods/f095ec3f-46ec-4b9a-a88e-fd7f1c183d05/volumes"
Sep 15 19:14:10.684789 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:10.684740    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8999-67119e" podUID="327231f9-795b-4074-be6c-3e3408c3ff89" containerName="test-curl-8999-67119e" containerID="cri-o://1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2" gracePeriod=2
Sep 15 19:14:10.853539 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:10.853519    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-672542"
Sep 15 19:14:10.973053 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:10.972986    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rqlx\" (UniqueName: \"kubernetes.io/projected/5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8-kube-api-access-2rqlx\") pod \"5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8\" (UID: \"5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8\") "
Sep 15 19:14:10.974952 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:10.974923    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8-kube-api-access-2rqlx" (OuterVolumeSpecName: "kube-api-access-2rqlx") pod "5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8" (UID: "5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8"). InnerVolumeSpecName "kube-api-access-2rqlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:11.074324 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:11.074299    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rqlx\" (UniqueName: \"kubernetes.io/projected/5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8-kube-api-access-2rqlx\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:14:11.689919 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:11.689880    2578 generic.go:356] "Generic (PLEG): container finished" podID="5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8" containerID="5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5" exitCode=137
Sep 15 19:14:11.690409 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:11.689924    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-672542"
Sep 15 19:14:11.690409 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:11.689981    2578 scope.go:117] "RemoveContainer" containerID="5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5"
Sep 15 19:14:11.699097 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:11.699082    2578 scope.go:117] "RemoveContainer" containerID="5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5"
Sep 15 19:14:11.699327 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:11.699306    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5\": container with ID starting with 5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5 not found: ID does not exist" containerID="5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5"
Sep 15 19:14:11.699367 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:11.699335    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5"} err="failed to get container status \"5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5\": rpc error: code = NotFound desc = could not find container \"5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5\": container with ID starting with 5ad6e4ece819fdc6c04c4a20807981776b797629e885f1c9f245db2fed8aa6d5 not found: ID does not exist"
Sep 15 19:14:11.700240 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:11.700219    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8999-672542\" is forbidden: User \"system:node:ip-10-0-135-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-221.ec2.internal' and this object" podUID="5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8" pod="opendatahub/test-curl-8999-672542"
Sep 15 19:14:12.098851 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:12.098749    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8" path="/var/lib/kubelet/pods/5d4077ab-ecfe-4dc0-a3cc-812ef9f76bc8/volumes"
Sep 15 19:14:12.866984 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:12.866960    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-67119e"
Sep 15 19:14:12.991447 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:12.991369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9dxz\" (UniqueName: \"kubernetes.io/projected/327231f9-795b-4074-be6c-3e3408c3ff89-kube-api-access-m9dxz\") pod \"327231f9-795b-4074-be6c-3e3408c3ff89\" (UID: \"327231f9-795b-4074-be6c-3e3408c3ff89\") "
Sep 15 19:14:12.993353 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:12.993328    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327231f9-795b-4074-be6c-3e3408c3ff89-kube-api-access-m9dxz" (OuterVolumeSpecName: "kube-api-access-m9dxz") pod "327231f9-795b-4074-be6c-3e3408c3ff89" (UID: "327231f9-795b-4074-be6c-3e3408c3ff89"). InnerVolumeSpecName "kube-api-access-m9dxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:13.092781 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:13.092742    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9dxz\" (UniqueName: \"kubernetes.io/projected/327231f9-795b-4074-be6c-3e3408c3ff89-kube-api-access-m9dxz\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:14:13.701879 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:13.701836    2578 generic.go:356] "Generic (PLEG): container finished" podID="327231f9-795b-4074-be6c-3e3408c3ff89" containerID="1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2" exitCode=137
Sep 15 19:14:13.702065 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:13.701885    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8999-67119e"
Sep 15 19:14:13.702065 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:13.701930    2578 scope.go:117] "RemoveContainer" containerID="1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2"
Sep 15 19:14:13.715560 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:13.715541    2578 scope.go:117] "RemoveContainer" containerID="1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2"
Sep 15 19:14:13.715856 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:13.715838    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2\": container with ID starting with 1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2 not found: ID does not exist" containerID="1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2"
Sep 15 19:14:13.715897 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:13.715864    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2"} err="failed to get container status \"1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2\": rpc error: code = NotFound desc = could not find container \"1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2\": container with ID starting with 1f5c5d84c1007e6d301b791bd316fd8c9b6825ebe3c42e38385119db0d2034c2 not found: ID does not exist"
Sep 15 19:14:13.716905 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:13.716886    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8999-67119e\" is forbidden: User \"system:node:ip-10-0-135-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-221.ec2.internal' and this object" podUID="327231f9-795b-4074-be6c-3e3408c3ff89" pod="opendatahub/test-curl-8999-67119e"
Sep 15 19:14:14.099313 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:14.099218    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="327231f9-795b-4074-be6c-3e3408c3ff89" path="/var/lib/kubelet/pods/327231f9-795b-4074-be6c-3e3408c3ff89/volumes"
Sep 15 19:14:15.686563 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:15.686529    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:14:19.892170 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:19.892134    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"]
Sep 15 19:14:19.892690 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:19.892675    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="327231f9-795b-4074-be6c-3e3408c3ff89" containerName="test-curl-8999-67119e"
Sep 15 19:14:19.892734 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:19.892693    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="327231f9-795b-4074-be6c-3e3408c3ff89" containerName="test-curl-8999-67119e"
Sep 15 19:14:19.892834 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:19.892820    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="327231f9-795b-4074-be6c-3e3408c3ff89" containerName="test-curl-8999-67119e"
Sep 15 19:14:19.897638 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:19.897617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:19.899789 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:19.899768    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-17c562de-serving-cert\""
Sep 15 19:14:19.905581 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:19.905558    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"]
Sep 15 19:14:20.051965 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.051931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-maas-api-tls\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.052127 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.051981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-metrics-certs\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.052127 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.051999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7lsr\" (UniqueName: \"kubernetes.io/projected/e4196617-7ee6-4d1d-9854-22222ec21b13-kube-api-access-q7lsr\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.052127 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.052044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4196617-7ee6-4d1d-9854-22222ec21b13-openshift-service-ca\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.152868 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.152768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-maas-api-tls\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.152868 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.152833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-metrics-certs\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.152868 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.152850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7lsr\" (UniqueName: \"kubernetes.io/projected/e4196617-7ee6-4d1d-9854-22222ec21b13-kube-api-access-q7lsr\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.153141 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.152878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4196617-7ee6-4d1d-9854-22222ec21b13-openshift-service-ca\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.153141 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:20.152941    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-serving-cert: secret "maas-api-e2e-mt-17c562de-serving-cert" not found
Sep 15 19:14:20.153141 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:20.153023    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-maas-api-tls podName:e4196617-7ee6-4d1d-9854-22222ec21b13 nodeName:}" failed. No retries permitted until 2026-09-15 19:14:20.653002607 +0000 UTC m=+1411.195098114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-maas-api-tls") pod "maas-api-e2e-mt-17c562de-86c6d468cd-vnchr" (UID: "e4196617-7ee6-4d1d-9854-22222ec21b13") : secret "maas-api-e2e-mt-17c562de-serving-cert" not found
Sep 15 19:14:20.153510 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.153489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4196617-7ee6-4d1d-9854-22222ec21b13-openshift-service-ca\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.155247 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.155224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-metrics-certs\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.163074 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.163051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7lsr\" (UniqueName: \"kubernetes.io/projected/e4196617-7ee6-4d1d-9854-22222ec21b13-kube-api-access-q7lsr\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.657463 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.657427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-maas-api-tls\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.659735 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.659716    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-maas-api-tls\") pod \"maas-api-e2e-mt-17c562de-86c6d468cd-vnchr\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.670966 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.670941    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"]
Sep 15 19:14:20.674349 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.674334    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:14:20.676358 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.676338    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-17c562de-dockercfg-wzjcr\""
Sep 15 19:14:20.676490 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.676474    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-17c562de\""
Sep 15 19:14:20.685227 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.685205    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"]
Sep 15 19:14:20.757922 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.757887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfea8da-84fd-4238-8d48-8f453958b782-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g\" (UID: \"ecfea8da-84fd-4238-8d48-8f453958b782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:14:20.758078 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.758022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlkdl\" (UniqueName: \"kubernetes.io/projected/ecfea8da-84fd-4238-8d48-8f453958b782-kube-api-access-qlkdl\") pod \"payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g\" (UID: \"ecfea8da-84fd-4238-8d48-8f453958b782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:14:20.809791 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.809756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:20.859146 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.859115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlkdl\" (UniqueName: \"kubernetes.io/projected/ecfea8da-84fd-4238-8d48-8f453958b782-kube-api-access-qlkdl\") pod \"payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g\" (UID: \"ecfea8da-84fd-4238-8d48-8f453958b782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:14:20.859274 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.859203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfea8da-84fd-4238-8d48-8f453958b782-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g\" (UID: \"ecfea8da-84fd-4238-8d48-8f453958b782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:14:20.860468 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.860441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfea8da-84fd-4238-8d48-8f453958b782-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g\" (UID: \"ecfea8da-84fd-4238-8d48-8f453958b782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:14:20.867483 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.867452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlkdl\" (UniqueName: \"kubernetes.io/projected/ecfea8da-84fd-4238-8d48-8f453958b782-kube-api-access-qlkdl\") pod \"payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g\" (UID: \"ecfea8da-84fd-4238-8d48-8f453958b782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:14:20.985262 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:20.985185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:14:21.079583 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:21.079560    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"]
Sep 15 19:14:21.081709 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:14:21.081681    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4196617_7ee6_4d1d_9854_22222ec21b13.slice/crio-5943fbbdf8d240a1d8d20dc169e31ae3a2eeda49392c56376df24fa383b6f651 WatchSource:0}: Error finding container 5943fbbdf8d240a1d8d20dc169e31ae3a2eeda49392c56376df24fa383b6f651: Status 404 returned error can't find the container with id 5943fbbdf8d240a1d8d20dc169e31ae3a2eeda49392c56376df24fa383b6f651
Sep 15 19:14:21.254198 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:21.254174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"]
Sep 15 19:14:21.256302 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:14:21.256272    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfea8da_84fd_4238_8d48_8f453958b782.slice/crio-44a6cff9ed29d0ec877a86cadbfea7778d80723cc8e685c3146d58678a33231d WatchSource:0}: Error finding container 44a6cff9ed29d0ec877a86cadbfea7778d80723cc8e685c3146d58678a33231d: Status 404 returned error can't find the container with id 44a6cff9ed29d0ec877a86cadbfea7778d80723cc8e685c3146d58678a33231d
Sep 15 19:14:21.740705 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:21.740666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g" event={"ID":"ecfea8da-84fd-4238-8d48-8f453958b782","Type":"ContainerStarted","Data":"bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049"}
Sep 15 19:14:21.740943 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:21.740728    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g" event={"ID":"ecfea8da-84fd-4238-8d48-8f453958b782","Type":"ContainerStarted","Data":"44a6cff9ed29d0ec877a86cadbfea7778d80723cc8e685c3146d58678a33231d"}
Sep 15 19:14:21.740943 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:21.740830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:14:21.742399 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:21.742370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr" event={"ID":"e4196617-7ee6-4d1d-9854-22222ec21b13","Type":"ContainerStarted","Data":"ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31"}
Sep 15 19:14:21.742513 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:21.742421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr" event={"ID":"e4196617-7ee6-4d1d-9854-22222ec21b13","Type":"ContainerStarted","Data":"5943fbbdf8d240a1d8d20dc169e31ae3a2eeda49392c56376df24fa383b6f651"}
Sep 15 19:14:21.742513 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:21.742445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:21.756525 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:21.756480    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g" podStartSLOduration=1.75646705 podStartE2EDuration="1.75646705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:14:21.755204216 +0000 UTC m=+1412.297299746" watchObservedRunningTime="2026-09-15 19:14:21.75646705 +0000 UTC m=+1412.298562575"
Sep 15 19:14:21.772079 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:21.772035    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr" podStartSLOduration=2.486450959 podStartE2EDuration="2.772024889s" totalImagesPullingTime="285.57393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:14:21.770301829 +0000 UTC m=+1412.312397362" watchObservedRunningTime="2026-09-15 19:14:21.772024889 +0000 UTC m=+1412.314120454"
Sep 15 19:14:27.752327 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:27.752293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:14:32.749597 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:32.749563    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:14:34.792028 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:34.791988    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"]
Sep 15 19:14:34.792520 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:34.792287    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8" podUID="9ecfbe08-fd43-4fdf-9079-651189163fd3" containerName="payload-processing" containerID="cri-o://11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52" gracePeriod=30
Sep 15 19:14:35.063609 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.063587    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:14:35.085379 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.085351    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ecfbe08-fd43-4fdf-9079-651189163fd3-plugins-config-volume\") pod \"9ecfbe08-fd43-4fdf-9079-651189163fd3\" (UID: \"9ecfbe08-fd43-4fdf-9079-651189163fd3\") "
Sep 15 19:14:35.085537 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.085447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57p7q\" (UniqueName: \"kubernetes.io/projected/9ecfbe08-fd43-4fdf-9079-651189163fd3-kube-api-access-57p7q\") pod \"9ecfbe08-fd43-4fdf-9079-651189163fd3\" (UID: \"9ecfbe08-fd43-4fdf-9079-651189163fd3\") "
Sep 15 19:14:35.085759 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.085735    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ecfbe08-fd43-4fdf-9079-651189163fd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ecfbe08-fd43-4fdf-9079-651189163fd3" (UID: "9ecfbe08-fd43-4fdf-9079-651189163fd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:14:35.087505 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.087479    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecfbe08-fd43-4fdf-9079-651189163fd3-kube-api-access-57p7q" (OuterVolumeSpecName: "kube-api-access-57p7q") pod "9ecfbe08-fd43-4fdf-9079-651189163fd3" (UID: "9ecfbe08-fd43-4fdf-9079-651189163fd3"). InnerVolumeSpecName "kube-api-access-57p7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:35.186222 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.186188    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ecfbe08-fd43-4fdf-9079-651189163fd3-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:14:35.186222 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.186221    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57p7q\" (UniqueName: \"kubernetes.io/projected/9ecfbe08-fd43-4fdf-9079-651189163fd3-kube-api-access-57p7q\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:14:35.823563 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.823531    2578 generic.go:356] "Generic (PLEG): container finished" podID="9ecfbe08-fd43-4fdf-9079-651189163fd3" containerID="11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52" exitCode=0
Sep 15 19:14:35.824021 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.823580    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"
Sep 15 19:14:35.824021 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.823611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8" event={"ID":"9ecfbe08-fd43-4fdf-9079-651189163fd3","Type":"ContainerDied","Data":"11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52"}
Sep 15 19:14:35.824021 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.823664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8" event={"ID":"9ecfbe08-fd43-4fdf-9079-651189163fd3","Type":"ContainerDied","Data":"f5d8cf962f8246a746478f63f7d0b548eacb498c5568c362c8b0817310af559b"}
Sep 15 19:14:35.824021 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.823686    2578 scope.go:117] "RemoveContainer" containerID="11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52"
Sep 15 19:14:35.832620 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.832604    2578 scope.go:117] "RemoveContainer" containerID="11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52"
Sep 15 19:14:35.832938 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:14:35.832919    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52\": container with ID starting with 11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52 not found: ID does not exist" containerID="11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52"
Sep 15 19:14:35.833009 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.832951    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52"} err="failed to get container status \"11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52\": rpc error: code = NotFound desc = could not find container \"11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52\": container with ID starting with 11026f6594836b59400e6c82de439a8da97872373162e49568fc218f008c5a52 not found: ID does not exist"
Sep 15 19:14:35.850295 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.850265    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"]
Sep 15 19:14:35.859691 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:35.859666    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9de44d-6546994bb9-f2qw8"]
Sep 15 19:14:36.104059 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:36.100410    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ecfbe08-fd43-4fdf-9079-651189163fd3" path="/var/lib/kubelet/pods/9ecfbe08-fd43-4fdf-9079-651189163fd3/volumes"
Sep 15 19:14:54.877493 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:14:54.877451    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:15:00.152846 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:00.152812    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2"]
Sep 15 19:15:00.153299 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:00.153178    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ecfbe08-fd43-4fdf-9079-651189163fd3" containerName="payload-processing"
Sep 15 19:15:00.153299 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:00.153190    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ecfbe08-fd43-4fdf-9079-651189163fd3" containerName="payload-processing"
Sep 15 19:15:00.153299 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:00.153250    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ecfbe08-fd43-4fdf-9079-651189163fd3" containerName="payload-processing"
Sep 15 19:15:00.156494 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:00.156477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2"
Sep 15 19:15:00.158956 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:00.158932    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:15:00.166619 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:00.166594    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2"]
Sep 15 19:15:00.167005 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:00.166987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2"
Sep 15 19:15:00.443577 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:00.443368    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2"]
Sep 15 19:15:00.445820 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:15:00.445773    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef46326_cef6_4169_839a_0ec4c0d646e8.slice/crio-788066c8171c1e13c3843cdc8afc366c3dd81eccf5d0a45e521c2f797f84803d WatchSource:0}: Error finding container 788066c8171c1e13c3843cdc8afc366c3dd81eccf5d0a45e521c2f797f84803d: Status 404 returned error can't find the container with id 788066c8171c1e13c3843cdc8afc366c3dd81eccf5d0a45e521c2f797f84803d
Sep 15 19:15:00.941624 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:00.941593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2" event={"ID":"5ef46326-cef6-4169-839a-0ec4c0d646e8","Type":"ContainerStarted","Data":"788066c8171c1e13c3843cdc8afc366c3dd81eccf5d0a45e521c2f797f84803d"}
Sep 15 19:15:02.954313 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:02.954278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2" event={"ID":"5ef46326-cef6-4169-839a-0ec4c0d646e8","Type":"ContainerStarted","Data":"92f1f48d406ad3673e5a4274b46b4ba12ceaa776203b5c59576b93b8dcb4d3df"}
Sep 15 19:15:02.968592 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:02.968496    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2" podStartSLOduration=0.983596983 podStartE2EDuration="2.968481903s" totalImagesPullingTime="1.98488492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:02.967604229 +0000 UTC m=+1453.509699757" watchObservedRunningTime="2026-09-15 19:15:02.968481903 +0000 UTC m=+1453.510577431"
Sep 15 19:15:03.961019 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:03.960977    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ef46326-cef6-4169-839a-0ec4c0d646e8" containerID="92f1f48d406ad3673e5a4274b46b4ba12ceaa776203b5c59576b93b8dcb4d3df" exitCode=0
Sep 15 19:15:03.961416 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:03.961028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2" event={"ID":"5ef46326-cef6-4169-839a-0ec4c0d646e8","Type":"ContainerDied","Data":"92f1f48d406ad3673e5a4274b46b4ba12ceaa776203b5c59576b93b8dcb4d3df"}
Sep 15 19:15:05.095205 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:05.095180    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2"
Sep 15 19:15:05.972349 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:05.972315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2" event={"ID":"5ef46326-cef6-4169-839a-0ec4c0d646e8","Type":"ContainerDied","Data":"788066c8171c1e13c3843cdc8afc366c3dd81eccf5d0a45e521c2f797f84803d"}
Sep 15 19:15:05.972600 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:05.972391    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="788066c8171c1e13c3843cdc8afc366c3dd81eccf5d0a45e521c2f797f84803d"
Sep 15 19:15:05.972600 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:05.972325    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824995-7nhr2"
Sep 15 19:15:09.665588 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:09.665557    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"]
Sep 15 19:15:16.585675 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:16.585637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:15:24.684712 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:24.684651    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j" podUID="c399bda2-751f-41db-a9e8-094f54a104cd" containerName="main" containerID="cri-o://644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf" gracePeriod=45
Sep 15 19:15:24.934183 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:24.934160    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:15:25.060894 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.060790    2578 generic.go:356] "Generic (PLEG): container finished" podID="c399bda2-751f-41db-a9e8-094f54a104cd" containerID="644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf" exitCode=0
Sep 15 19:15:25.060894 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.060856    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"
Sep 15 19:15:25.060894 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.060874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j" event={"ID":"c399bda2-751f-41db-a9e8-094f54a104cd","Type":"ContainerDied","Data":"644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf"}
Sep 15 19:15:25.061116 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.060920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j" event={"ID":"c399bda2-751f-41db-a9e8-094f54a104cd","Type":"ContainerDied","Data":"697b52c42936eb4cf0b22ae815ee7037c5fdefb2e5ab95aeb6248deb6515c963"}
Sep 15 19:15:25.061116 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.060942    2578 scope.go:117] "RemoveContainer" containerID="644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf"
Sep 15 19:15:25.068962 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.068947    2578 scope.go:117] "RemoveContainer" containerID="644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf"
Sep 15 19:15:25.069230 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:15:25.069210    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf\": container with ID starting with 644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf not found: ID does not exist" containerID="644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf"
Sep 15 19:15:25.069277 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.069239    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf"} err="failed to get container status \"644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf\": rpc error: code = NotFound desc = could not find container \"644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf\": container with ID starting with 644139a83cb60044202716dcd5df29ce3817ba967e32141bf7301c2e1d0d8ccf not found: ID does not exist"
Sep 15 19:15:25.113649 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.113624    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-model-cache\") pod \"c399bda2-751f-41db-a9e8-094f54a104cd\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") "
Sep 15 19:15:25.113751 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.113679    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8sq8\" (UniqueName: \"kubernetes.io/projected/c399bda2-751f-41db-a9e8-094f54a104cd-kube-api-access-p8sq8\") pod \"c399bda2-751f-41db-a9e8-094f54a104cd\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") "
Sep 15 19:15:25.113751 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.113718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-tmp-dir\") pod \"c399bda2-751f-41db-a9e8-094f54a104cd\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") "
Sep 15 19:15:25.113751 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.113750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-dshm\") pod \"c399bda2-751f-41db-a9e8-094f54a104cd\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") "
Sep 15 19:15:25.113897 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.113787    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c399bda2-751f-41db-a9e8-094f54a104cd-tls-certs\") pod \"c399bda2-751f-41db-a9e8-094f54a104cd\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") "
Sep 15 19:15:25.113897 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.113834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-home\") pod \"c399bda2-751f-41db-a9e8-094f54a104cd\" (UID: \"c399bda2-751f-41db-a9e8-094f54a104cd\") "
Sep 15 19:15:25.113996 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.113894    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "c399bda2-751f-41db-a9e8-094f54a104cd" (UID: "c399bda2-751f-41db-a9e8-094f54a104cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:25.114078 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.114053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c399bda2-751f-41db-a9e8-094f54a104cd" (UID: "c399bda2-751f-41db-a9e8-094f54a104cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:25.114179 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.114096    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-model-cache\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:15:25.114179 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.114134    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-home" (OuterVolumeSpecName: "home") pod "c399bda2-751f-41db-a9e8-094f54a104cd" (UID: "c399bda2-751f-41db-a9e8-094f54a104cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:25.115955 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.115930    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-dshm" (OuterVolumeSpecName: "dshm") pod "c399bda2-751f-41db-a9e8-094f54a104cd" (UID: "c399bda2-751f-41db-a9e8-094f54a104cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:25.116066 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.115969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c399bda2-751f-41db-a9e8-094f54a104cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c399bda2-751f-41db-a9e8-094f54a104cd" (UID: "c399bda2-751f-41db-a9e8-094f54a104cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:15:25.116066 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.115987    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c399bda2-751f-41db-a9e8-094f54a104cd-kube-api-access-p8sq8" (OuterVolumeSpecName: "kube-api-access-p8sq8") pod "c399bda2-751f-41db-a9e8-094f54a104cd" (UID: "c399bda2-751f-41db-a9e8-094f54a104cd"). InnerVolumeSpecName "kube-api-access-p8sq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:25.215200 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.215172    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8sq8\" (UniqueName: \"kubernetes.io/projected/c399bda2-751f-41db-a9e8-094f54a104cd-kube-api-access-p8sq8\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:15:25.215200 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.215198    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-tmp-dir\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:15:25.215387 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.215210    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-dshm\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:15:25.215387 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.215218    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c399bda2-751f-41db-a9e8-094f54a104cd-tls-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:15:25.215387 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.215225    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c399bda2-751f-41db-a9e8-094f54a104cd-home\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:15:25.382117 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.382080    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"]
Sep 15 19:15:25.385973 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:25.385948    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3daa58/test-model-3daa58-kserve-7c4684df65-fhm5j"]
Sep 15 19:15:26.098628 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:26.098586    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c399bda2-751f-41db-a9e8-094f54a104cd" path="/var/lib/kubelet/pods/c399bda2-751f-41db-a9e8-094f54a104cd/volumes"
Sep 15 19:15:28.664966 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:28.664931    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"]
Sep 15 19:15:28.665392 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:28.665138    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct" podUID="4759ff24-1def-4fdd-b519-113c28254f04" containerName="payload-processing" containerID="cri-o://a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90" gracePeriod=30
Sep 15 19:15:28.699918 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:28.699871    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"]
Sep 15 19:15:28.700510 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:28.700470    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj" podUID="03ff452a-fd2d-4f06-873e-c1a98c94cb3b" containerName="payload-pre-processing" containerID="cri-o://6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff" gracePeriod=30
Sep 15 19:15:28.976689 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:28.976664    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:15:28.979361 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:28.979340    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:15:29.046636 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.046603    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg7fx\" (UniqueName: \"kubernetes.io/projected/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-kube-api-access-jg7fx\") pod \"03ff452a-fd2d-4f06-873e-c1a98c94cb3b\" (UID: \"03ff452a-fd2d-4f06-873e-c1a98c94cb3b\") "
Sep 15 19:15:29.046860 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.046642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d6pp\" (UniqueName: \"kubernetes.io/projected/4759ff24-1def-4fdd-b519-113c28254f04-kube-api-access-9d6pp\") pod \"4759ff24-1def-4fdd-b519-113c28254f04\" (UID: \"4759ff24-1def-4fdd-b519-113c28254f04\") "
Sep 15 19:15:29.046860 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.046691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-plugins-config-volume\") pod \"03ff452a-fd2d-4f06-873e-c1a98c94cb3b\" (UID: \"03ff452a-fd2d-4f06-873e-c1a98c94cb3b\") "
Sep 15 19:15:29.046860 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.046733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4759ff24-1def-4fdd-b519-113c28254f04-plugins-config-volume\") pod \"4759ff24-1def-4fdd-b519-113c28254f04\" (UID: \"4759ff24-1def-4fdd-b519-113c28254f04\") "
Sep 15 19:15:29.047144 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.047114    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03ff452a-fd2d-4f06-873e-c1a98c94cb3b" (UID: "03ff452a-fd2d-4f06-873e-c1a98c94cb3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:15:29.047350 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.047309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4759ff24-1def-4fdd-b519-113c28254f04-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4759ff24-1def-4fdd-b519-113c28254f04" (UID: "4759ff24-1def-4fdd-b519-113c28254f04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:15:29.049129 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.049088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-kube-api-access-jg7fx" (OuterVolumeSpecName: "kube-api-access-jg7fx") pod "03ff452a-fd2d-4f06-873e-c1a98c94cb3b" (UID: "03ff452a-fd2d-4f06-873e-c1a98c94cb3b"). InnerVolumeSpecName "kube-api-access-jg7fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:29.049431 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.049397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4759ff24-1def-4fdd-b519-113c28254f04-kube-api-access-9d6pp" (OuterVolumeSpecName: "kube-api-access-9d6pp") pod "4759ff24-1def-4fdd-b519-113c28254f04" (UID: "4759ff24-1def-4fdd-b519-113c28254f04"). InnerVolumeSpecName "kube-api-access-9d6pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:29.082773 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.082747    2578 generic.go:356] "Generic (PLEG): container finished" podID="4759ff24-1def-4fdd-b519-113c28254f04" containerID="a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90" exitCode=0
Sep 15 19:15:29.082936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.082782    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"
Sep 15 19:15:29.082936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.082867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct" event={"ID":"4759ff24-1def-4fdd-b519-113c28254f04","Type":"ContainerDied","Data":"a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90"}
Sep 15 19:15:29.082936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.082925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct" event={"ID":"4759ff24-1def-4fdd-b519-113c28254f04","Type":"ContainerDied","Data":"5c027ff4755cfffb6a9d4d46a4584d6923e49c21da31fa0d49aa3235bccaad08"}
Sep 15 19:15:29.083110 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.082955    2578 scope.go:117] "RemoveContainer" containerID="a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90"
Sep 15 19:15:29.084176 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.084153    2578 generic.go:356] "Generic (PLEG): container finished" podID="03ff452a-fd2d-4f06-873e-c1a98c94cb3b" containerID="6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff" exitCode=0
Sep 15 19:15:29.084255 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.084200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj" event={"ID":"03ff452a-fd2d-4f06-873e-c1a98c94cb3b","Type":"ContainerDied","Data":"6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff"}
Sep 15 19:15:29.084255 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.084234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj" event={"ID":"03ff452a-fd2d-4f06-873e-c1a98c94cb3b","Type":"ContainerDied","Data":"507509f4edd7d64a96ca4e5c8b60f26c78ce2f36ca70634f345bc3c60c7825ef"}
Sep 15 19:15:29.084255 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.084196    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"
Sep 15 19:15:29.092419 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.092313    2578 scope.go:117] "RemoveContainer" containerID="a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90"
Sep 15 19:15:29.092688 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:15:29.092670    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90\": container with ID starting with a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90 not found: ID does not exist" containerID="a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90"
Sep 15 19:15:29.092770 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.092695    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90"} err="failed to get container status \"a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90\": rpc error: code = NotFound desc = could not find container \"a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90\": container with ID starting with a34ed4b91dcecb1099e80ac1ae27fbcc6fce0507fb1bc7bbb43d571440db4a90 not found: ID does not exist"
Sep 15 19:15:29.092770 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.092710    2578 scope.go:117] "RemoveContainer" containerID="6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff"
Sep 15 19:15:29.100453 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.100435    2578 scope.go:117] "RemoveContainer" containerID="6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff"
Sep 15 19:15:29.100685 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:15:29.100668    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff\": container with ID starting with 6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff not found: ID does not exist" containerID="6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff"
Sep 15 19:15:29.100732 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.100691    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff"} err="failed to get container status \"6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff\": rpc error: code = NotFound desc = could not find container \"6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff\": container with ID starting with 6e10020332291ab9c90f3790c73426f99d019d5014b3acc6f5442c88f899e8ff not found: ID does not exist"
Sep 15 19:15:29.122768 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.122739    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"]
Sep 15 19:15:29.133164 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.133141    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3daa58-cbfb8b887-s2bct"]
Sep 15 19:15:29.147514 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.147489    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg7fx\" (UniqueName: \"kubernetes.io/projected/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-kube-api-access-jg7fx\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:15:29.147514 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.147509    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6pp\" (UniqueName: \"kubernetes.io/projected/4759ff24-1def-4fdd-b519-113c28254f04-kube-api-access-9d6pp\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:15:29.147640 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.147518    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ff452a-fd2d-4f06-873e-c1a98c94cb3b-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:15:29.147640 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.147526    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4759ff24-1def-4fdd-b519-113c28254f04-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:15:29.155947 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.155922    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"]
Sep 15 19:15:29.159987 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:29.159968    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3daa58-9dc576f5d-xwjtj"]
Sep 15 19:15:30.099027 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:30.098950    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03ff452a-fd2d-4f06-873e-c1a98c94cb3b" path="/var/lib/kubelet/pods/03ff452a-fd2d-4f06-873e-c1a98c94cb3b/volumes"
Sep 15 19:15:30.099372 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:30.099359    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4759ff24-1def-4fdd-b519-113c28254f04" path="/var/lib/kubelet/pods/4759ff24-1def-4fdd-b519-113c28254f04/volumes"
Sep 15 19:15:43.240838 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.240779    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"]
Sep 15 19:15:43.241417 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241384    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef46326-cef6-4169-839a-0ec4c0d646e8" containerName="cleanup"
Sep 15 19:15:43.241417 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241405    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef46326-cef6-4169-839a-0ec4c0d646e8" containerName="cleanup"
Sep 15 19:15:43.241542 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241422    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4759ff24-1def-4fdd-b519-113c28254f04" containerName="payload-processing"
Sep 15 19:15:43.241542 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241432    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4759ff24-1def-4fdd-b519-113c28254f04" containerName="payload-processing"
Sep 15 19:15:43.241542 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241441    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c399bda2-751f-41db-a9e8-094f54a104cd" containerName="main"
Sep 15 19:15:43.241542 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241451    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c399bda2-751f-41db-a9e8-094f54a104cd" containerName="main"
Sep 15 19:15:43.241542 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241461    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ff452a-fd2d-4f06-873e-c1a98c94cb3b" containerName="payload-pre-processing"
Sep 15 19:15:43.241542 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241470    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ff452a-fd2d-4f06-873e-c1a98c94cb3b" containerName="payload-pre-processing"
Sep 15 19:15:43.241871 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241554    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4759ff24-1def-4fdd-b519-113c28254f04" containerName="payload-processing"
Sep 15 19:15:43.241871 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241570    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef46326-cef6-4169-839a-0ec4c0d646e8" containerName="cleanup"
Sep 15 19:15:43.241871 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241580    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c399bda2-751f-41db-a9e8-094f54a104cd" containerName="main"
Sep 15 19:15:43.241871 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.241590    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ff452a-fd2d-4f06-873e-c1a98c94cb3b" containerName="payload-pre-processing"
Sep 15 19:15:43.249098 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.249071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.251706 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.251684    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c6d10a-serving-cert\""
Sep 15 19:15:43.257428 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.257405    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"]
Sep 15 19:15:43.373935 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.373902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.374143 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.373947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c12e64d5-57a8-4045-9bd3-f840ef58a075-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.374143 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.374028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqtfs\" (UniqueName: \"kubernetes.io/projected/c12e64d5-57a8-4045-9bd3-f840ef58a075-kube-api-access-wqtfs\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.374143 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.374103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.475272 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.475232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqtfs\" (UniqueName: \"kubernetes.io/projected/c12e64d5-57a8-4045-9bd3-f840ef58a075-kube-api-access-wqtfs\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.475272 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.475286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.475541 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.475348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.475541 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.475372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c12e64d5-57a8-4045-9bd3-f840ef58a075-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.476039 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.476019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c12e64d5-57a8-4045-9bd3-f840ef58a075-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.477835 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.477816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.477946 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.477925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.482372 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.482349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqtfs\" (UniqueName: \"kubernetes.io/projected/c12e64d5-57a8-4045-9bd3-f840ef58a075-kube-api-access-wqtfs\") pod \"maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.561598 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.561498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:43.834889 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.834859    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"]
Sep 15 19:15:43.837522 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:15:43.837490    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12e64d5_57a8_4045_9bd3_f840ef58a075.slice/crio-5a670e56623787b4f40387aad84f8e5042945829fd24cd3f58e68e896276cf84 WatchSource:0}: Error finding container 5a670e56623787b4f40387aad84f8e5042945829fd24cd3f58e68e896276cf84: Status 404 returned error can't find the container with id 5a670e56623787b4f40387aad84f8e5042945829fd24cd3f58e68e896276cf84
Sep 15 19:15:43.865139 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.865112    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"]
Sep 15 19:15:43.871955 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.870870    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:15:43.873711 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.873686    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c6d10a-dockercfg-gjh5f\""
Sep 15 19:15:43.874035 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.874020    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c6d10a\""
Sep 15 19:15:43.878696 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.878672    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"]
Sep 15 19:15:43.979739 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.979695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx\" (UID: \"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:15:43.979922 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:43.979758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzjt8\" (UniqueName: \"kubernetes.io/projected/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-kube-api-access-fzjt8\") pod \"payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx\" (UID: \"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:15:44.081054 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:44.081018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx\" (UID: \"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:15:44.081221 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:44.081077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzjt8\" (UniqueName: \"kubernetes.io/projected/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-kube-api-access-fzjt8\") pod \"payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx\" (UID: \"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:15:44.081700 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:44.081679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx\" (UID: \"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:15:44.089378 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:44.089318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzjt8\" (UniqueName: \"kubernetes.io/projected/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-kube-api-access-fzjt8\") pod \"payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx\" (UID: \"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:15:44.155877 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:44.155838    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df" event={"ID":"c12e64d5-57a8-4045-9bd3-f840ef58a075","Type":"ContainerStarted","Data":"5a670e56623787b4f40387aad84f8e5042945829fd24cd3f58e68e896276cf84"}
Sep 15 19:15:44.184125 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:44.184097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:15:44.453485 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:44.453448    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"]
Sep 15 19:15:44.456377 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:15:44.456349    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad802fa5_0cd3_4ad0_bfd3_49bdd88f2daa.slice/crio-ea15ef49dcef577ea3614300e1ac0828e7590f8f07965eed357536391d0beed1 WatchSource:0}: Error finding container ea15ef49dcef577ea3614300e1ac0828e7590f8f07965eed357536391d0beed1: Status 404 returned error can't find the container with id ea15ef49dcef577ea3614300e1ac0828e7590f8f07965eed357536391d0beed1
Sep 15 19:15:45.162135 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:45.162094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df" event={"ID":"c12e64d5-57a8-4045-9bd3-f840ef58a075","Type":"ContainerStarted","Data":"14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d"}
Sep 15 19:15:45.162341 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:45.162212    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:45.163656 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:45.163630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx" event={"ID":"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa","Type":"ContainerStarted","Data":"cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced"}
Sep 15 19:15:45.163776 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:45.163678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx" event={"ID":"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa","Type":"ContainerStarted","Data":"ea15ef49dcef577ea3614300e1ac0828e7590f8f07965eed357536391d0beed1"}
Sep 15 19:15:45.163776 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:45.163716    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:15:45.181617 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:45.181568    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df" podStartSLOduration=1.492186605 podStartE2EDuration="2.181551139s" totalImagesPullingTime="689.364534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:45.179340362 +0000 UTC m=+1495.721435891" watchObservedRunningTime="2026-09-15 19:15:45.181551139 +0000 UTC m=+1495.723646687"
Sep 15 19:15:45.194301 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:45.194250    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx" podStartSLOduration=2.194232197 podStartE2EDuration="2.194232197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:45.193568968 +0000 UTC m=+1495.735664496" watchObservedRunningTime="2026-09-15 19:15:45.194232197 +0000 UTC m=+1495.736327722"
Sep 15 19:15:51.173630 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:51.173598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:15:53.881911 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:53.881869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:15:56.171067 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:15:56.171036    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:16:18.187625 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.187547    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"]
Sep 15 19:16:18.190954 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.190938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.193667 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.193646    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-415f87-serving-cert\""
Sep 15 19:16:18.200377 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.200327    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"]
Sep 15 19:16:18.282146 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.282115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a6f1f4-2497-4960-9054-7275a473c292-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.282300 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.282185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.282300 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.282223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.282300 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.282248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsqm\" (UniqueName: \"kubernetes.io/projected/15a6f1f4-2497-4960-9054-7275a473c292-kube-api-access-dhsqm\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.383392 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.383360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.383392 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.383393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsqm\" (UniqueName: \"kubernetes.io/projected/15a6f1f4-2497-4960-9054-7275a473c292-kube-api-access-dhsqm\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.383624 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.383433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a6f1f4-2497-4960-9054-7275a473c292-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.383624 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.383480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.383624 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:16:18.383558    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-serving-cert: secret "maas-api-e2e-shared-b-w2-415f87-serving-cert" not found
Sep 15 19:16:18.383624 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:16:18.383614    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-maas-api-tls podName:15a6f1f4-2497-4960-9054-7275a473c292 nodeName:}" failed. No retries permitted until 2026-09-15 19:16:18.883599401 +0000 UTC m=+1529.425694921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-maas-api-tls") pod "maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd" (UID: "15a6f1f4-2497-4960-9054-7275a473c292") : secret "maas-api-e2e-shared-b-w2-415f87-serving-cert" not found
Sep 15 19:16:18.384182 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.384162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a6f1f4-2497-4960-9054-7275a473c292-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.385600 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.385575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.393122 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.393103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsqm\" (UniqueName: \"kubernetes.io/projected/15a6f1f4-2497-4960-9054-7275a473c292-kube-api-access-dhsqm\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.885958 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.885924    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"]
Sep 15 19:16:18.888160 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.888139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.889692 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.889673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:16:18.890520 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.890502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:18.892428 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.892412    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-415f87-dockercfg-7lwnq\""
Sep 15 19:16:18.892767 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.892750    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-415f87\""
Sep 15 19:16:18.902975 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.902954    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"]
Sep 15 19:16:18.988739 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.988707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea37e1d3-a803-4f98-93d4-589ebbc59e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh\" (UID: \"ea37e1d3-a803-4f98-93d4-589ebbc59e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:16:18.988906 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:18.988753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g8gh\" (UniqueName: \"kubernetes.io/projected/ea37e1d3-a803-4f98-93d4-589ebbc59e42-kube-api-access-9g8gh\") pod \"payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh\" (UID: \"ea37e1d3-a803-4f98-93d4-589ebbc59e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:16:19.089936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:19.089903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea37e1d3-a803-4f98-93d4-589ebbc59e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh\" (UID: \"ea37e1d3-a803-4f98-93d4-589ebbc59e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:16:19.090107 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:19.089951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g8gh\" (UniqueName: \"kubernetes.io/projected/ea37e1d3-a803-4f98-93d4-589ebbc59e42-kube-api-access-9g8gh\") pod \"payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh\" (UID: \"ea37e1d3-a803-4f98-93d4-589ebbc59e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:16:19.090564 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:19.090544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea37e1d3-a803-4f98-93d4-589ebbc59e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh\" (UID: \"ea37e1d3-a803-4f98-93d4-589ebbc59e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:16:19.097418 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:19.097384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g8gh\" (UniqueName: \"kubernetes.io/projected/ea37e1d3-a803-4f98-93d4-589ebbc59e42-kube-api-access-9g8gh\") pod \"payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh\" (UID: \"ea37e1d3-a803-4f98-93d4-589ebbc59e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:16:19.103082 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:19.103064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:19.209188 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:19.209161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:16:19.371304 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:19.371269    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"]
Sep 15 19:16:19.374363 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:16:19.374330    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a6f1f4_2497_4960_9054_7275a473c292.slice/crio-fddd2eaafc51ea621eb272d592d5edcd44d3b41d17b5e2f4c170912228f71746 WatchSource:0}: Error finding container fddd2eaafc51ea621eb272d592d5edcd44d3b41d17b5e2f4c170912228f71746: Status 404 returned error can't find the container with id fddd2eaafc51ea621eb272d592d5edcd44d3b41d17b5e2f4c170912228f71746
Sep 15 19:16:19.477057 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:19.477034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"]
Sep 15 19:16:19.478518 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:16:19.478485    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea37e1d3_a803_4f98_93d4_589ebbc59e42.slice/crio-16c87cea3256500f7240c655cb569aba283fbdb55bbad3347b4e61f3a0803326 WatchSource:0}: Error finding container 16c87cea3256500f7240c655cb569aba283fbdb55bbad3347b4e61f3a0803326: Status 404 returned error can't find the container with id 16c87cea3256500f7240c655cb569aba283fbdb55bbad3347b4e61f3a0803326
Sep 15 19:16:20.330576 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:20.330538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh" event={"ID":"ea37e1d3-a803-4f98-93d4-589ebbc59e42","Type":"ContainerStarted","Data":"a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd"}
Sep 15 19:16:20.331083 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:20.330601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh" event={"ID":"ea37e1d3-a803-4f98-93d4-589ebbc59e42","Type":"ContainerStarted","Data":"16c87cea3256500f7240c655cb569aba283fbdb55bbad3347b4e61f3a0803326"}
Sep 15 19:16:20.331083 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:20.330760    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:16:20.332244 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:20.332215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd" event={"ID":"15a6f1f4-2497-4960-9054-7275a473c292","Type":"ContainerStarted","Data":"3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d"}
Sep 15 19:16:20.332420 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:20.332250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd" event={"ID":"15a6f1f4-2497-4960-9054-7275a473c292","Type":"ContainerStarted","Data":"fddd2eaafc51ea621eb272d592d5edcd44d3b41d17b5e2f4c170912228f71746"}
Sep 15 19:16:20.332420 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:20.332279    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:20.346606 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:20.346567    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh" podStartSLOduration=2.34655405 podStartE2EDuration="2.34655405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:16:20.344992959 +0000 UTC m=+1530.887088489" watchObservedRunningTime="2026-09-15 19:16:20.34655405 +0000 UTC m=+1530.888649578"
Sep 15 19:16:20.364423 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:20.364378    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd" podStartSLOduration=1.988484847 podStartE2EDuration="2.364361941s" totalImagesPullingTime="375.877094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:16:20.362385148 +0000 UTC m=+1530.904480676" watchObservedRunningTime="2026-09-15 19:16:20.364361941 +0000 UTC m=+1530.906457469"
Sep 15 19:16:26.342269 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:26.342237    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:16:29.881885 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:29.881844    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:16:31.339403 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:16:31.339372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:17:00.277612 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:00.277573    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:17:32.788870 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:32.788835    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:17:49.200557 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.200474    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"]
Sep 15 19:17:49.201076 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.200718    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr" podUID="e4196617-7ee6-4d1d-9854-22222ec21b13" containerName="maas-api" containerID="cri-o://ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31" gracePeriod=30
Sep 15 19:17:49.263272 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.263233    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"]
Sep 15 19:17:49.263550 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.263490    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g" podUID="ecfea8da-84fd-4238-8d48-8f453958b782" containerName="payload-pre-processing" containerID="cri-o://bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049" gracePeriod=30
Sep 15 19:17:49.509666 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.509399    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:17:49.545890 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.545857    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:17:49.593014 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.592978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfea8da-84fd-4238-8d48-8f453958b782-plugins-config-volume\") pod \"ecfea8da-84fd-4238-8d48-8f453958b782\" (UID: \"ecfea8da-84fd-4238-8d48-8f453958b782\") "
Sep 15 19:17:49.593170 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.593030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-maas-api-tls\") pod \"e4196617-7ee6-4d1d-9854-22222ec21b13\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") "
Sep 15 19:17:49.593170 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.593063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-metrics-certs\") pod \"e4196617-7ee6-4d1d-9854-22222ec21b13\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") "
Sep 15 19:17:49.593170 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.593141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4196617-7ee6-4d1d-9854-22222ec21b13-openshift-service-ca\") pod \"e4196617-7ee6-4d1d-9854-22222ec21b13\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") "
Sep 15 19:17:49.593170 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.593170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlkdl\" (UniqueName: \"kubernetes.io/projected/ecfea8da-84fd-4238-8d48-8f453958b782-kube-api-access-qlkdl\") pod \"ecfea8da-84fd-4238-8d48-8f453958b782\" (UID: \"ecfea8da-84fd-4238-8d48-8f453958b782\") "
Sep 15 19:17:49.593377 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.593199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7lsr\" (UniqueName: \"kubernetes.io/projected/e4196617-7ee6-4d1d-9854-22222ec21b13-kube-api-access-q7lsr\") pod \"e4196617-7ee6-4d1d-9854-22222ec21b13\" (UID: \"e4196617-7ee6-4d1d-9854-22222ec21b13\") "
Sep 15 19:17:49.593561 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.593525    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4196617-7ee6-4d1d-9854-22222ec21b13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4196617-7ee6-4d1d-9854-22222ec21b13" (UID: "e4196617-7ee6-4d1d-9854-22222ec21b13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:17:49.593561 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.593547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfea8da-84fd-4238-8d48-8f453958b782-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecfea8da-84fd-4238-8d48-8f453958b782" (UID: "ecfea8da-84fd-4238-8d48-8f453958b782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:17:49.595267 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.595231    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfea8da-84fd-4238-8d48-8f453958b782-kube-api-access-qlkdl" (OuterVolumeSpecName: "kube-api-access-qlkdl") pod "ecfea8da-84fd-4238-8d48-8f453958b782" (UID: "ecfea8da-84fd-4238-8d48-8f453958b782"). InnerVolumeSpecName "kube-api-access-qlkdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:49.595267 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.595240    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4196617-7ee6-4d1d-9854-22222ec21b13" (UID: "e4196617-7ee6-4d1d-9854-22222ec21b13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:49.595427 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.595319    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e4196617-7ee6-4d1d-9854-22222ec21b13" (UID: "e4196617-7ee6-4d1d-9854-22222ec21b13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:49.595427 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.595319    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4196617-7ee6-4d1d-9854-22222ec21b13-kube-api-access-q7lsr" (OuterVolumeSpecName: "kube-api-access-q7lsr") pod "e4196617-7ee6-4d1d-9854-22222ec21b13" (UID: "e4196617-7ee6-4d1d-9854-22222ec21b13"). InnerVolumeSpecName "kube-api-access-q7lsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:49.694168 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.694132    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfea8da-84fd-4238-8d48-8f453958b782-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:17:49.694168 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.694164    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:17:49.694168 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.694173    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4196617-7ee6-4d1d-9854-22222ec21b13-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:17:49.694418 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.694183    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4196617-7ee6-4d1d-9854-22222ec21b13-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:17:49.694418 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.694192    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlkdl\" (UniqueName: \"kubernetes.io/projected/ecfea8da-84fd-4238-8d48-8f453958b782-kube-api-access-qlkdl\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:17:49.694418 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.694202    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7lsr\" (UniqueName: \"kubernetes.io/projected/e4196617-7ee6-4d1d-9854-22222ec21b13-kube-api-access-q7lsr\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:17:49.764223 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.764115    2578 generic.go:356] "Generic (PLEG): container finished" podID="ecfea8da-84fd-4238-8d48-8f453958b782" containerID="bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049" exitCode=0
Sep 15 19:17:49.764223 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.764167    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"
Sep 15 19:17:49.764223 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.764207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g" event={"ID":"ecfea8da-84fd-4238-8d48-8f453958b782","Type":"ContainerDied","Data":"bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049"}
Sep 15 19:17:49.764500 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.764277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g" event={"ID":"ecfea8da-84fd-4238-8d48-8f453958b782","Type":"ContainerDied","Data":"44a6cff9ed29d0ec877a86cadbfea7778d80723cc8e685c3146d58678a33231d"}
Sep 15 19:17:49.764500 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.764304    2578 scope.go:117] "RemoveContainer" containerID="bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049"
Sep 15 19:17:49.765643 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.765621    2578 generic.go:356] "Generic (PLEG): container finished" podID="e4196617-7ee6-4d1d-9854-22222ec21b13" containerID="ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31" exitCode=0
Sep 15 19:17:49.765743 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.765664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr" event={"ID":"e4196617-7ee6-4d1d-9854-22222ec21b13","Type":"ContainerDied","Data":"ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31"}
Sep 15 19:17:49.765743 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.765683    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"
Sep 15 19:17:49.765743 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.765702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr" event={"ID":"e4196617-7ee6-4d1d-9854-22222ec21b13","Type":"ContainerDied","Data":"5943fbbdf8d240a1d8d20dc169e31ae3a2eeda49392c56376df24fa383b6f651"}
Sep 15 19:17:49.774005 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.773972    2578 scope.go:117] "RemoveContainer" containerID="bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049"
Sep 15 19:17:49.774257 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:17:49.774239    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049\": container with ID starting with bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049 not found: ID does not exist" containerID="bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049"
Sep 15 19:17:49.774336 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.774264    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049"} err="failed to get container status \"bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049\": rpc error: code = NotFound desc = could not find container \"bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049\": container with ID starting with bb40f52851d3df1efe622d4d881463526f1fe1ea605401af5ebfb5facefc3049 not found: ID does not exist"
Sep 15 19:17:49.774336 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.774279    2578 scope.go:117] "RemoveContainer" containerID="ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31"
Sep 15 19:17:49.782254 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.782234    2578 scope.go:117] "RemoveContainer" containerID="ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31"
Sep 15 19:17:49.782503 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:17:49.782482    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31\": container with ID starting with ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31 not found: ID does not exist" containerID="ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31"
Sep 15 19:17:49.782568 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.782514    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31"} err="failed to get container status \"ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31\": rpc error: code = NotFound desc = could not find container \"ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31\": container with ID starting with ee30f3177d594db6b0bf1f49d26a9ceec5ba7aeec83c7d7a20127cd99b7c8f31 not found: ID does not exist"
Sep 15 19:17:49.785769 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.785745    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"]
Sep 15 19:17:49.789520 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.789499    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17c562de-6ccbf8f74-cm77g"]
Sep 15 19:17:49.799564 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.799540    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"]
Sep 15 19:17:49.805342 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:49.805321    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17c562de-86c6d468cd-vnchr"]
Sep 15 19:17:50.099480 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.099396    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4196617-7ee6-4d1d-9854-22222ec21b13" path="/var/lib/kubelet/pods/e4196617-7ee6-4d1d-9854-22222ec21b13/volumes"
Sep 15 19:17:50.099848 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.099827    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecfea8da-84fd-4238-8d48-8f453958b782" path="/var/lib/kubelet/pods/ecfea8da-84fd-4238-8d48-8f453958b782/volumes"
Sep 15 19:17:50.439606 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.439571    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"]
Sep 15 19:17:50.440084 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.439864    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w" podUID="f1c72779-4ea8-4ef2-8344-a9c266150251" containerName="maas-api" containerID="cri-o://5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760" gracePeriod=30
Sep 15 19:17:50.683183 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.683157    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:17:50.771240 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.771154    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1c72779-4ea8-4ef2-8344-a9c266150251" containerID="5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760" exitCode=0
Sep 15 19:17:50.771240 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.771192    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"
Sep 15 19:17:50.771425 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.771241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w" event={"ID":"f1c72779-4ea8-4ef2-8344-a9c266150251","Type":"ContainerDied","Data":"5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760"}
Sep 15 19:17:50.771425 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.771291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w" event={"ID":"f1c72779-4ea8-4ef2-8344-a9c266150251","Type":"ContainerDied","Data":"06052d88de18e1ce89e24b29b8b2514858847eb5f004a390b9fd7e536eac005f"}
Sep 15 19:17:50.771425 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.771318    2578 scope.go:117] "RemoveContainer" containerID="5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760"
Sep 15 19:17:50.779673 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.779656    2578 scope.go:117] "RemoveContainer" containerID="5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760"
Sep 15 19:17:50.779973 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:17:50.779947    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760\": container with ID starting with 5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760 not found: ID does not exist" containerID="5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760"
Sep 15 19:17:50.780041 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.779981    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760"} err="failed to get container status \"5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760\": rpc error: code = NotFound desc = could not find container \"5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760\": container with ID starting with 5b63b0fce9d7c1f373a7745ab79cafb6873e6cffc1ed24086fb1052014f60760 not found: ID does not exist"
Sep 15 19:17:50.804325 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.804298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djn7g\" (UniqueName: \"kubernetes.io/projected/f1c72779-4ea8-4ef2-8344-a9c266150251-kube-api-access-djn7g\") pod \"f1c72779-4ea8-4ef2-8344-a9c266150251\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") "
Sep 15 19:17:50.804431 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.804364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-metrics-certs\") pod \"f1c72779-4ea8-4ef2-8344-a9c266150251\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") "
Sep 15 19:17:50.804431 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.804427    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c72779-4ea8-4ef2-8344-a9c266150251-openshift-service-ca\") pod \"f1c72779-4ea8-4ef2-8344-a9c266150251\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") "
Sep 15 19:17:50.804525 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.804458    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-maas-api-tls\") pod \"f1c72779-4ea8-4ef2-8344-a9c266150251\" (UID: \"f1c72779-4ea8-4ef2-8344-a9c266150251\") "
Sep 15 19:17:50.804790 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.804755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c72779-4ea8-4ef2-8344-a9c266150251-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f1c72779-4ea8-4ef2-8344-a9c266150251" (UID: "f1c72779-4ea8-4ef2-8344-a9c266150251"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:17:50.806443 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.806417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c72779-4ea8-4ef2-8344-a9c266150251-kube-api-access-djn7g" (OuterVolumeSpecName: "kube-api-access-djn7g") pod "f1c72779-4ea8-4ef2-8344-a9c266150251" (UID: "f1c72779-4ea8-4ef2-8344-a9c266150251"). InnerVolumeSpecName "kube-api-access-djn7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:50.806918 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.806901    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f1c72779-4ea8-4ef2-8344-a9c266150251" (UID: "f1c72779-4ea8-4ef2-8344-a9c266150251"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:50.806980 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.806922    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f1c72779-4ea8-4ef2-8344-a9c266150251" (UID: "f1c72779-4ea8-4ef2-8344-a9c266150251"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:50.906209 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.906170    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djn7g\" (UniqueName: \"kubernetes.io/projected/f1c72779-4ea8-4ef2-8344-a9c266150251-kube-api-access-djn7g\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:17:50.906209 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.906211    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:17:50.906405 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.906225    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c72779-4ea8-4ef2-8344-a9c266150251-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:17:50.906405 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:50.906239    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1c72779-4ea8-4ef2-8344-a9c266150251-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:17:51.092579 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:51.092553    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"]
Sep 15 19:17:51.095668 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:51.095639    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f0ad35-66d4f4f9f4-ddw9w"]
Sep 15 19:17:52.104510 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:17:52.104475    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c72779-4ea8-4ef2-8344-a9c266150251" path="/var/lib/kubelet/pods/f1c72779-4ea8-4ef2-8344-a9c266150251/volumes"
Sep 15 19:18:14.689005 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:18:14.688965    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:18:40.973946 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:18:40.973908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:19:11.681668 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:11.681632    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:19:37.773347 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:37.773244    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:19:46.795257 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.795224    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"]
Sep 15 19:19:46.795721 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.795577    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecfea8da-84fd-4238-8d48-8f453958b782" containerName="payload-pre-processing"
Sep 15 19:19:46.795721 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.795588    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecfea8da-84fd-4238-8d48-8f453958b782" containerName="payload-pre-processing"
Sep 15 19:19:46.795721 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.795606    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4196617-7ee6-4d1d-9854-22222ec21b13" containerName="maas-api"
Sep 15 19:19:46.795721 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.795611    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4196617-7ee6-4d1d-9854-22222ec21b13" containerName="maas-api"
Sep 15 19:19:46.795721 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.795622    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c72779-4ea8-4ef2-8344-a9c266150251" containerName="maas-api"
Sep 15 19:19:46.795721 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.795627    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c72779-4ea8-4ef2-8344-a9c266150251" containerName="maas-api"
Sep 15 19:19:46.795721 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.795686    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c72779-4ea8-4ef2-8344-a9c266150251" containerName="maas-api"
Sep 15 19:19:46.795721 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.795695    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecfea8da-84fd-4238-8d48-8f453958b782" containerName="payload-pre-processing"
Sep 15 19:19:46.795721 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.795702    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4196617-7ee6-4d1d-9854-22222ec21b13" containerName="maas-api"
Sep 15 19:19:46.798854 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.798830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.801264 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.801243    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-759580-serving-cert\""
Sep 15 19:19:46.809811 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.809776    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"]
Sep 15 19:19:46.883049 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.883015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2dg\" (UniqueName: \"kubernetes.io/projected/51b1d9a0-2fc4-445b-afc5-97b084e41947-kube-api-access-sz2dg\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.883210 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.883079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-metrics-certs\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.883210 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.883104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-maas-api-tls\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.883210 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.883153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b1d9a0-2fc4-445b-afc5-97b084e41947-openshift-service-ca\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.983841 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.983791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b1d9a0-2fc4-445b-afc5-97b084e41947-openshift-service-ca\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.984000 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.983883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2dg\" (UniqueName: \"kubernetes.io/projected/51b1d9a0-2fc4-445b-afc5-97b084e41947-kube-api-access-sz2dg\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.984000 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.983948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-metrics-certs\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.984000 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.983973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-maas-api-tls\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.984590 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.984565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b1d9a0-2fc4-445b-afc5-97b084e41947-openshift-service-ca\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.986441 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.986421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-maas-api-tls\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.986634 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.986612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-metrics-certs\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:46.992078 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:46.992056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2dg\" (UniqueName: \"kubernetes.io/projected/51b1d9a0-2fc4-445b-afc5-97b084e41947-kube-api-access-sz2dg\") pod \"maas-api-e2e-api-a-759580-8c94547f6-xx5lt\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:47.111526 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:47.111451    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:47.384111 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:47.384086    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"]
Sep 15 19:19:47.389089 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:19:47.389060    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b1d9a0_2fc4_445b_afc5_97b084e41947.slice/crio-82791b12f1c9264006efe7f97677b56048cfa7c865339175598af1fcc58b4391 WatchSource:0}: Error finding container 82791b12f1c9264006efe7f97677b56048cfa7c865339175598af1fcc58b4391: Status 404 returned error can't find the container with id 82791b12f1c9264006efe7f97677b56048cfa7c865339175598af1fcc58b4391
Sep 15 19:19:47.391076 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:47.391055    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:19:48.321067 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:48.320963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt" event={"ID":"51b1d9a0-2fc4-445b-afc5-97b084e41947","Type":"ContainerStarted","Data":"dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8"}
Sep 15 19:19:48.321067 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:48.321056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt" event={"ID":"51b1d9a0-2fc4-445b-afc5-97b084e41947","Type":"ContainerStarted","Data":"82791b12f1c9264006efe7f97677b56048cfa7c865339175598af1fcc58b4391"}
Sep 15 19:19:48.321567 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:48.321103    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:19:48.342888 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:48.342839    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt" podStartSLOduration=1.851124363 podStartE2EDuration="2.342825356s" totalImagesPullingTime="491.700993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:19:48.340432138 +0000 UTC m=+1738.882527687" watchObservedRunningTime="2026-09-15 19:19:48.342825356 +0000 UTC m=+1738.884920883"
Sep 15 19:19:54.331871 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:19:54.331841    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:20:01.382184 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:01.382143    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"]
Sep 15 19:20:06.275171 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:06.275131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:20:11.808790 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:11.808755    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"]
Sep 15 19:20:11.812391 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:11.812368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:11.814516 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:11.814493    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-16a967-serving-cert\""
Sep 15 19:20:11.822044 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:11.822022    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"]
Sep 15 19:20:11.905273 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:11.905245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppwwx\" (UniqueName: \"kubernetes.io/projected/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-kube-api-access-ppwwx\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:11.905432 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:11.905316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-maas-api-tls\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:11.905432 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:11.905341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-metrics-certs\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:11.905432 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:11.905360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-openshift-service-ca\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:12.006116 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.006079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-maas-api-tls\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:12.006295 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.006128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-metrics-certs\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:12.006295 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.006159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-openshift-service-ca\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:12.006295 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.006208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppwwx\" (UniqueName: \"kubernetes.io/projected/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-kube-api-access-ppwwx\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:12.006924 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.006902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-openshift-service-ca\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:12.008517 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.008493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-maas-api-tls\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:12.008622 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.008564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-metrics-certs\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:12.013898 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.013872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppwwx\" (UniqueName: \"kubernetes.io/projected/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-kube-api-access-ppwwx\") pod \"maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:12.124820 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.124739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:12.404586 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.404435    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"]
Sep 15 19:20:12.406987 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:20:12.406959    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c2ed7d_e638_4f27_b6b6_aa61ab876b42.slice/crio-98d8cab8d878a47d7fda8869364a369767ea1fd90c9f691243b8c63a2f86a4f6 WatchSource:0}: Error finding container 98d8cab8d878a47d7fda8869364a369767ea1fd90c9f691243b8c63a2f86a4f6: Status 404 returned error can't find the container with id 98d8cab8d878a47d7fda8869364a369767ea1fd90c9f691243b8c63a2f86a4f6
Sep 15 19:20:12.418531 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.418507    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"]
Sep 15 19:20:12.422985 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.422967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:20:12.425241 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.425223    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-16a967-dockercfg-989tg\""
Sep 15 19:20:12.425349 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.425278    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-16a967\""
Sep 15 19:20:12.428880 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.428858    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"]
Sep 15 19:20:12.438310 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.438287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr" event={"ID":"12c2ed7d-e638-4f27-b6b6-aa61ab876b42","Type":"ContainerStarted","Data":"98d8cab8d878a47d7fda8869364a369767ea1fd90c9f691243b8c63a2f86a4f6"}
Sep 15 19:20:12.510494 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.510462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjvn\" (UniqueName: \"kubernetes.io/projected/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-kube-api-access-fkjvn\") pod \"payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc\" (UID: \"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a\") " pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:20:12.510638 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.510499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc\" (UID: \"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a\") " pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:20:12.601624 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.601584    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"]
Sep 15 19:20:12.601900 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.601852    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2" podUID="dad6ea98-8625-427e-ad4e-5fb504ff8dd0" containerName="payload-processing" containerID="cri-o://309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346" gracePeriod=30
Sep 15 19:20:12.605679 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.605650    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"]
Sep 15 19:20:12.605955 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.605881    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk" podUID="f7954e7c-224c-4f41-aed2-d68b31550af8" containerName="payload-pre-processing" containerID="cri-o://267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78" gracePeriod=30
Sep 15 19:20:12.611344 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.611275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjvn\" (UniqueName: \"kubernetes.io/projected/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-kube-api-access-fkjvn\") pod \"payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc\" (UID: \"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a\") " pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:20:12.611344 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.611323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc\" (UID: \"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a\") " pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:20:12.612141 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.612095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc\" (UID: \"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a\") " pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:20:12.624157 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.624133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjvn\" (UniqueName: \"kubernetes.io/projected/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-kube-api-access-fkjvn\") pod \"payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc\" (UID: \"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a\") " pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:20:12.736141 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.736104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:20:12.861664 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.861642    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:20:12.894929 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.894901    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:20:12.914585 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.914506    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-plugins-config-volume\") pod \"dad6ea98-8625-427e-ad4e-5fb504ff8dd0\" (UID: \"dad6ea98-8625-427e-ad4e-5fb504ff8dd0\") "
Sep 15 19:20:12.914748 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.914601    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbj45\" (UniqueName: \"kubernetes.io/projected/f7954e7c-224c-4f41-aed2-d68b31550af8-kube-api-access-pbj45\") pod \"f7954e7c-224c-4f41-aed2-d68b31550af8\" (UID: \"f7954e7c-224c-4f41-aed2-d68b31550af8\") "
Sep 15 19:20:12.914748 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.914637    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7954e7c-224c-4f41-aed2-d68b31550af8-plugins-config-volume\") pod \"f7954e7c-224c-4f41-aed2-d68b31550af8\" (UID: \"f7954e7c-224c-4f41-aed2-d68b31550af8\") "
Sep 15 19:20:12.914748 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.914728    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c76ld\" (UniqueName: \"kubernetes.io/projected/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-kube-api-access-c76ld\") pod \"dad6ea98-8625-427e-ad4e-5fb504ff8dd0\" (UID: \"dad6ea98-8625-427e-ad4e-5fb504ff8dd0\") "
Sep 15 19:20:12.915218 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.915179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dad6ea98-8625-427e-ad4e-5fb504ff8dd0" (UID: "dad6ea98-8625-427e-ad4e-5fb504ff8dd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:12.915334 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.915216    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7954e7c-224c-4f41-aed2-d68b31550af8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7954e7c-224c-4f41-aed2-d68b31550af8" (UID: "f7954e7c-224c-4f41-aed2-d68b31550af8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:12.917485 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.917462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7954e7c-224c-4f41-aed2-d68b31550af8-kube-api-access-pbj45" (OuterVolumeSpecName: "kube-api-access-pbj45") pod "f7954e7c-224c-4f41-aed2-d68b31550af8" (UID: "f7954e7c-224c-4f41-aed2-d68b31550af8"). InnerVolumeSpecName "kube-api-access-pbj45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:12.917585 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:12.917530    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-kube-api-access-c76ld" (OuterVolumeSpecName: "kube-api-access-c76ld") pod "dad6ea98-8625-427e-ad4e-5fb504ff8dd0" (UID: "dad6ea98-8625-427e-ad4e-5fb504ff8dd0"). InnerVolumeSpecName "kube-api-access-c76ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:13.014634 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.014559    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"]
Sep 15 19:20:13.015613 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.015594    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c76ld\" (UniqueName: \"kubernetes.io/projected/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-kube-api-access-c76ld\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:13.015707 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.015616    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad6ea98-8625-427e-ad4e-5fb504ff8dd0-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:13.015707 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.015631    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbj45\" (UniqueName: \"kubernetes.io/projected/f7954e7c-224c-4f41-aed2-d68b31550af8-kube-api-access-pbj45\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:13.015707 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.015644    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7954e7c-224c-4f41-aed2-d68b31550af8-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:13.017453 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:20:13.017410    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8aedf5a_46fb_4e8a_bd1c_6e0ea6b6179a.slice/crio-8f15f8e8ddc9b98bc082c33042360f18cbfab8878bdb6c13b3018d2facb2c7b2 WatchSource:0}: Error finding container 8f15f8e8ddc9b98bc082c33042360f18cbfab8878bdb6c13b3018d2facb2c7b2: Status 404 returned error can't find the container with id 8f15f8e8ddc9b98bc082c33042360f18cbfab8878bdb6c13b3018d2facb2c7b2
Sep 15 19:20:13.445477 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.445430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr" event={"ID":"12c2ed7d-e638-4f27-b6b6-aa61ab876b42","Type":"ContainerStarted","Data":"c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de"}
Sep 15 19:20:13.445676 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.445523    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:13.446935 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.446912    2578 generic.go:356] "Generic (PLEG): container finished" podID="dad6ea98-8625-427e-ad4e-5fb504ff8dd0" containerID="309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346" exitCode=0
Sep 15 19:20:13.447056 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.446935    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"
Sep 15 19:20:13.447056 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.446998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2" event={"ID":"dad6ea98-8625-427e-ad4e-5fb504ff8dd0","Type":"ContainerDied","Data":"309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346"}
Sep 15 19:20:13.447056 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.447032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2" event={"ID":"dad6ea98-8625-427e-ad4e-5fb504ff8dd0","Type":"ContainerDied","Data":"7372d8f0d31658b360a06259184a1281fadea4dbf9bf422ff0e1037a7d2809e3"}
Sep 15 19:20:13.447205 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.447063    2578 scope.go:117] "RemoveContainer" containerID="309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346"
Sep 15 19:20:13.449204 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.449179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc" event={"ID":"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a","Type":"ContainerStarted","Data":"c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4"}
Sep 15 19:20:13.449342 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.449224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc" event={"ID":"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a","Type":"ContainerStarted","Data":"8f15f8e8ddc9b98bc082c33042360f18cbfab8878bdb6c13b3018d2facb2c7b2"}
Sep 15 19:20:13.449560 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.449539    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:20:13.450648 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.450623    2578 generic.go:356] "Generic (PLEG): container finished" podID="f7954e7c-224c-4f41-aed2-d68b31550af8" containerID="267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78" exitCode=0
Sep 15 19:20:13.450761 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.450648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk" event={"ID":"f7954e7c-224c-4f41-aed2-d68b31550af8","Type":"ContainerDied","Data":"267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78"}
Sep 15 19:20:13.450761 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.450701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk" event={"ID":"f7954e7c-224c-4f41-aed2-d68b31550af8","Type":"ContainerDied","Data":"36cd649850326b6fcbb2438221a23f0434be81ec1912fb6c26c176ac661f659e"}
Sep 15 19:20:13.450761 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.450658    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"
Sep 15 19:20:13.464101 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.464046    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr" podStartSLOduration=2.075643417 podStartE2EDuration="2.464030061s" totalImagesPullingTime="388.386644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:20:13.463848955 +0000 UTC m=+1764.005944486" watchObservedRunningTime="2026-09-15 19:20:13.464030061 +0000 UTC m=+1764.006125592"
Sep 15 19:20:13.465500 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.465475    2578 scope.go:117] "RemoveContainer" containerID="309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346"
Sep 15 19:20:13.465876 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:20:13.465845    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346\": container with ID starting with 309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346 not found: ID does not exist" containerID="309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346"
Sep 15 19:20:13.466001 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.465886    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346"} err="failed to get container status \"309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346\": rpc error: code = NotFound desc = could not find container \"309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346\": container with ID starting with 309f4a5f5d2d34111ddbb3289e48b40bb6fcd5ac77b40d26053fc92e37d40346 not found: ID does not exist"
Sep 15 19:20:13.466001 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.465912    2578 scope.go:117] "RemoveContainer" containerID="267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78"
Sep 15 19:20:13.475169 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.475148    2578 scope.go:117] "RemoveContainer" containerID="267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78"
Sep 15 19:20:13.475456 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:20:13.475432    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78\": container with ID starting with 267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78 not found: ID does not exist" containerID="267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78"
Sep 15 19:20:13.475517 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.475467    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78"} err="failed to get container status \"267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78\": rpc error: code = NotFound desc = could not find container \"267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78\": container with ID starting with 267de7e80a61137acd8804dc5f2572a248628bf10594a336bd9feb1e93339d78 not found: ID does not exist"
Sep 15 19:20:13.480887 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.480866    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"]
Sep 15 19:20:13.483253 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.483228    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-900c88-76d74c4495-h9zf2"]
Sep 15 19:20:13.492552 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:20:13.492532    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad6ea98_8625_427e_ad4e_5fb504ff8dd0.slice\": RecentStats: unable to find data in memory cache]"
Sep 15 19:20:13.499560 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.499523    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc" podStartSLOduration=1.499510787 podStartE2EDuration="1.499510787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:20:13.498118793 +0000 UTC m=+1764.040214323" watchObservedRunningTime="2026-09-15 19:20:13.499510787 +0000 UTC m=+1764.041606316"
Sep 15 19:20:13.510371 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.510347    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"]
Sep 15 19:20:13.513891 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:13.513869    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-900c88-5fddc98c7d-ttpqk"]
Sep 15 19:20:14.102076 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:14.102046    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad6ea98-8625-427e-ad4e-5fb504ff8dd0" path="/var/lib/kubelet/pods/dad6ea98-8625-427e-ad4e-5fb504ff8dd0/volumes"
Sep 15 19:20:14.102461 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:14.102392    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7954e7c-224c-4f41-aed2-d68b31550af8" path="/var/lib/kubelet/pods/f7954e7c-224c-4f41-aed2-d68b31550af8/volumes"
Sep 15 19:20:16.401529 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.401480    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6" podUID="33fda78e-3d3b-4288-8368-f7e7d0c518f2" containerName="main" containerID="cri-o://10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a" gracePeriod=45
Sep 15 19:20:16.651678 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.651621    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:20:16.746098 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.746063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-dshm\") pod \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") "
Sep 15 19:20:16.746286 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.746132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tmp-dir\") pod \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") "
Sep 15 19:20:16.746286 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.746170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-home\") pod \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") "
Sep 15 19:20:16.746286 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.746197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-model-cache\") pod \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") "
Sep 15 19:20:16.746286 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.746270    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tls-certs\") pod \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") "
Sep 15 19:20:16.746514 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.746305    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzlrg\" (UniqueName: \"kubernetes.io/projected/33fda78e-3d3b-4288-8368-f7e7d0c518f2-kube-api-access-tzlrg\") pod \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\" (UID: \"33fda78e-3d3b-4288-8368-f7e7d0c518f2\") "
Sep 15 19:20:16.746514 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.746448    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33fda78e-3d3b-4288-8368-f7e7d0c518f2" (UID: "33fda78e-3d3b-4288-8368-f7e7d0c518f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.746621 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.746570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "33fda78e-3d3b-4288-8368-f7e7d0c518f2" (UID: "33fda78e-3d3b-4288-8368-f7e7d0c518f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.746621 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.746600    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tmp-dir\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.746722 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.746685    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-home" (OuterVolumeSpecName: "home") pod "33fda78e-3d3b-4288-8368-f7e7d0c518f2" (UID: "33fda78e-3d3b-4288-8368-f7e7d0c518f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.748721 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.748692    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33fda78e-3d3b-4288-8368-f7e7d0c518f2" (UID: "33fda78e-3d3b-4288-8368-f7e7d0c518f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:16.748866 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.748719    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33fda78e-3d3b-4288-8368-f7e7d0c518f2-kube-api-access-tzlrg" (OuterVolumeSpecName: "kube-api-access-tzlrg") pod "33fda78e-3d3b-4288-8368-f7e7d0c518f2" (UID: "33fda78e-3d3b-4288-8368-f7e7d0c518f2"). InnerVolumeSpecName "kube-api-access-tzlrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:16.748866 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.748835    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-dshm" (OuterVolumeSpecName: "dshm") pod "33fda78e-3d3b-4288-8368-f7e7d0c518f2" (UID: "33fda78e-3d3b-4288-8368-f7e7d0c518f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.847226 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.847194    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-dshm\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.847226 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.847219    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-home\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.847226 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.847228    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fda78e-3d3b-4288-8368-f7e7d0c518f2-model-cache\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.847472 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.847238    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fda78e-3d3b-4288-8368-f7e7d0c518f2-tls-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.847472 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:16.847248    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzlrg\" (UniqueName: \"kubernetes.io/projected/33fda78e-3d3b-4288-8368-f7e7d0c518f2-kube-api-access-tzlrg\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:17.472697 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:17.472660    2578 generic.go:356] "Generic (PLEG): container finished" podID="33fda78e-3d3b-4288-8368-f7e7d0c518f2" containerID="10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a" exitCode=0
Sep 15 19:20:17.473187 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:17.472704    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"
Sep 15 19:20:17.473187 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:17.472716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6" event={"ID":"33fda78e-3d3b-4288-8368-f7e7d0c518f2","Type":"ContainerDied","Data":"10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a"}
Sep 15 19:20:17.473187 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:17.472755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6" event={"ID":"33fda78e-3d3b-4288-8368-f7e7d0c518f2","Type":"ContainerDied","Data":"ba8a157e709b7d64cf26f384145188a5189c63cc259191785f304706b537d941"}
Sep 15 19:20:17.473187 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:17.472776    2578 scope.go:117] "RemoveContainer" containerID="10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a"
Sep 15 19:20:17.481653 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:17.481634    2578 scope.go:117] "RemoveContainer" containerID="10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a"
Sep 15 19:20:17.481959 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:20:17.481941    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a\": container with ID starting with 10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a not found: ID does not exist" containerID="10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a"
Sep 15 19:20:17.482006 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:17.481973    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a"} err="failed to get container status \"10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a\": rpc error: code = NotFound desc = could not find container \"10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a\": container with ID starting with 10e959fdd600f53ba4f5804320944398b45ba6a41792162a8c796f677d3cb05a not found: ID does not exist"
Sep 15 19:20:17.492750 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:17.492727    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"]
Sep 15 19:20:17.496558 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:17.496538    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/distinct-w0-900c88-kserve-c858876c9-x9pf6"]
Sep 15 19:20:18.099346 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:18.099302    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33fda78e-3d3b-4288-8368-f7e7d0c518f2" path="/var/lib/kubelet/pods/33fda78e-3d3b-4288-8368-f7e7d0c518f2/volumes"
Sep 15 19:20:19.462594 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:19.462565    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:20:25.464422 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:25.464381    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:20:41.781946 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:41.781913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:20:55.025065 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.025030    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"]
Sep 15 19:20:55.025464 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.025238    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt" podUID="51b1d9a0-2fc4-445b-afc5-97b084e41947" containerName="maas-api" containerID="cri-o://dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8" gracePeriod=30
Sep 15 19:20:55.277110 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.277054    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:20:55.285601 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.285581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-maas-api-tls\") pod \"51b1d9a0-2fc4-445b-afc5-97b084e41947\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") "
Sep 15 19:20:55.285710 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.285634    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz2dg\" (UniqueName: \"kubernetes.io/projected/51b1d9a0-2fc4-445b-afc5-97b084e41947-kube-api-access-sz2dg\") pod \"51b1d9a0-2fc4-445b-afc5-97b084e41947\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") "
Sep 15 19:20:55.285710 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.285655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-metrics-certs\") pod \"51b1d9a0-2fc4-445b-afc5-97b084e41947\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") "
Sep 15 19:20:55.285853 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.285750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b1d9a0-2fc4-445b-afc5-97b084e41947-openshift-service-ca\") pod \"51b1d9a0-2fc4-445b-afc5-97b084e41947\" (UID: \"51b1d9a0-2fc4-445b-afc5-97b084e41947\") "
Sep 15 19:20:55.286200 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.286177    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b1d9a0-2fc4-445b-afc5-97b084e41947-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51b1d9a0-2fc4-445b-afc5-97b084e41947" (UID: "51b1d9a0-2fc4-445b-afc5-97b084e41947"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:55.287728 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.287698    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51b1d9a0-2fc4-445b-afc5-97b084e41947" (UID: "51b1d9a0-2fc4-445b-afc5-97b084e41947"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:55.287863 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.287743    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b1d9a0-2fc4-445b-afc5-97b084e41947-kube-api-access-sz2dg" (OuterVolumeSpecName: "kube-api-access-sz2dg") pod "51b1d9a0-2fc4-445b-afc5-97b084e41947" (UID: "51b1d9a0-2fc4-445b-afc5-97b084e41947"). InnerVolumeSpecName "kube-api-access-sz2dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:55.287863 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.287741    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "51b1d9a0-2fc4-445b-afc5-97b084e41947" (UID: "51b1d9a0-2fc4-445b-afc5-97b084e41947"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:55.386609 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.386573    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:55.386609 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.386609    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz2dg\" (UniqueName: \"kubernetes.io/projected/51b1d9a0-2fc4-445b-afc5-97b084e41947-kube-api-access-sz2dg\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:55.386609 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.386619    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b1d9a0-2fc4-445b-afc5-97b084e41947-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:55.386609 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.386627    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b1d9a0-2fc4-445b-afc5-97b084e41947-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:20:55.661883 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.661844    2578 generic.go:356] "Generic (PLEG): container finished" podID="51b1d9a0-2fc4-445b-afc5-97b084e41947" containerID="dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8" exitCode=0
Sep 15 19:20:55.662056 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.661888    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"
Sep 15 19:20:55.662056 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.661895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt" event={"ID":"51b1d9a0-2fc4-445b-afc5-97b084e41947","Type":"ContainerDied","Data":"dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8"}
Sep 15 19:20:55.662056 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.661944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt" event={"ID":"51b1d9a0-2fc4-445b-afc5-97b084e41947","Type":"ContainerDied","Data":"82791b12f1c9264006efe7f97677b56048cfa7c865339175598af1fcc58b4391"}
Sep 15 19:20:55.662056 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.661975    2578 scope.go:117] "RemoveContainer" containerID="dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8"
Sep 15 19:20:55.671633 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.671615    2578 scope.go:117] "RemoveContainer" containerID="dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8"
Sep 15 19:20:55.671914 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:20:55.671895    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8\": container with ID starting with dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8 not found: ID does not exist" containerID="dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8"
Sep 15 19:20:55.671991 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.671920    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8"} err="failed to get container status \"dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8\": rpc error: code = NotFound desc = could not find container \"dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8\": container with ID starting with dcf26df7382448f97073b05b9229f0e572a6a5e13d5c72c4b8ec98a39ba8e5e8 not found: ID does not exist"
Sep 15 19:20:55.681682 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.681654    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"]
Sep 15 19:20:55.685221 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:55.685200    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-759580-8c94547f6-xx5lt"]
Sep 15 19:20:56.099180 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:56.099102    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b1d9a0-2fc4-445b-afc5-97b084e41947" path="/var/lib/kubelet/pods/51b1d9a0-2fc4-445b-afc5-97b084e41947/volumes"
Sep 15 19:20:59.771681 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:20:59.771646    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:21:17.580951 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:17.580920    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:21:17.897544 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:17.897515    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"]
Sep 15 19:21:17.897787 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:17.897755    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr" podUID="12c2ed7d-e638-4f27-b6b6-aa61ab876b42" containerName="maas-api" containerID="cri-o://c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de" gracePeriod=30
Sep 15 19:21:17.951050 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:17.951013    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"]
Sep 15 19:21:17.951297 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:17.951269    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc" podUID="c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a" containerName="payload-processing" containerID="cri-o://c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4" gracePeriod=30
Sep 15 19:21:18.191185 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.191104    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:21:18.218270 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.218245    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:21:18.264936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.264910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-maas-api-tls\") pod \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") "
Sep 15 19:21:18.265103 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.264945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-openshift-service-ca\") pod \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") "
Sep 15 19:21:18.265103 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.265014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppwwx\" (UniqueName: \"kubernetes.io/projected/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-kube-api-access-ppwwx\") pod \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") "
Sep 15 19:21:18.265103 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.265042    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-metrics-certs\") pod \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\" (UID: \"12c2ed7d-e638-4f27-b6b6-aa61ab876b42\") "
Sep 15 19:21:18.265103 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.265070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-plugins-config-volume\") pod \"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a\" (UID: \"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a\") "
Sep 15 19:21:18.265344 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.265135    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkjvn\" (UniqueName: \"kubernetes.io/projected/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-kube-api-access-fkjvn\") pod \"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a\" (UID: \"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a\") "
Sep 15 19:21:18.265487 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.265406    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12c2ed7d-e638-4f27-b6b6-aa61ab876b42" (UID: "12c2ed7d-e638-4f27-b6b6-aa61ab876b42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:18.265842 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.265780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a" (UID: "c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:18.267469 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.267441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12c2ed7d-e638-4f27-b6b6-aa61ab876b42" (UID: "12c2ed7d-e638-4f27-b6b6-aa61ab876b42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:18.267609 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.267590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-kube-api-access-ppwwx" (OuterVolumeSpecName: "kube-api-access-ppwwx") pod "12c2ed7d-e638-4f27-b6b6-aa61ab876b42" (UID: "12c2ed7d-e638-4f27-b6b6-aa61ab876b42"). InnerVolumeSpecName "kube-api-access-ppwwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:18.267812 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.267777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-kube-api-access-fkjvn" (OuterVolumeSpecName: "kube-api-access-fkjvn") pod "c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a" (UID: "c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a"). InnerVolumeSpecName "kube-api-access-fkjvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:18.267870 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.267788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "12c2ed7d-e638-4f27-b6b6-aa61ab876b42" (UID: "12c2ed7d-e638-4f27-b6b6-aa61ab876b42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:18.366694 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.366667    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppwwx\" (UniqueName: \"kubernetes.io/projected/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-kube-api-access-ppwwx\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:18.366694 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.366692    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:18.366916 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.366702    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:18.366916 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.366712    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkjvn\" (UniqueName: \"kubernetes.io/projected/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a-kube-api-access-fkjvn\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:18.366916 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.366720    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:18.366916 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.366728    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c2ed7d-e638-4f27-b6b6-aa61ab876b42-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:18.773770 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.773732    2578 generic.go:356] "Generic (PLEG): container finished" podID="c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a" containerID="c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4" exitCode=0
Sep 15 19:21:18.774238 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.773778    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"
Sep 15 19:21:18.774238 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.773831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc" event={"ID":"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a","Type":"ContainerDied","Data":"c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4"}
Sep 15 19:21:18.774238 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.773889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc" event={"ID":"c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a","Type":"ContainerDied","Data":"8f15f8e8ddc9b98bc082c33042360f18cbfab8878bdb6c13b3018d2facb2c7b2"}
Sep 15 19:21:18.774238 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.773918    2578 scope.go:117] "RemoveContainer" containerID="c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4"
Sep 15 19:21:18.775732 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.775708    2578 generic.go:356] "Generic (PLEG): container finished" podID="12c2ed7d-e638-4f27-b6b6-aa61ab876b42" containerID="c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de" exitCode=0
Sep 15 19:21:18.775865 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.775739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr" event={"ID":"12c2ed7d-e638-4f27-b6b6-aa61ab876b42","Type":"ContainerDied","Data":"c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de"}
Sep 15 19:21:18.775865 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.775758    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"
Sep 15 19:21:18.775865 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.775783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr" event={"ID":"12c2ed7d-e638-4f27-b6b6-aa61ab876b42","Type":"ContainerDied","Data":"98d8cab8d878a47d7fda8869364a369767ea1fd90c9f691243b8c63a2f86a4f6"}
Sep 15 19:21:18.785376 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.785359    2578 scope.go:117] "RemoveContainer" containerID="c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4"
Sep 15 19:21:18.785626 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:21:18.785607    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4\": container with ID starting with c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4 not found: ID does not exist" containerID="c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4"
Sep 15 19:21:18.785675 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.785634    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4"} err="failed to get container status \"c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4\": rpc error: code = NotFound desc = could not find container \"c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4\": container with ID starting with c53644b9e19d7294335e074de1ee1c3b483f73b4a6d79442b67a0204f8fda3e4 not found: ID does not exist"
Sep 15 19:21:18.785675 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.785650    2578 scope.go:117] "RemoveContainer" containerID="c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de"
Sep 15 19:21:18.793484 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.793468    2578 scope.go:117] "RemoveContainer" containerID="c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de"
Sep 15 19:21:18.793724 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:21:18.793709    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de\": container with ID starting with c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de not found: ID does not exist" containerID="c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de"
Sep 15 19:21:18.793822 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.793729    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de"} err="failed to get container status \"c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de\": rpc error: code = NotFound desc = could not find container \"c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de\": container with ID starting with c52098c01463ac49b4bf45705aad3ef27ff1e7b338b4d564791909deadd896de not found: ID does not exist"
Sep 15 19:21:18.802116 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.802095    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"]
Sep 15 19:21:18.806342 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.806315    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-16a967-8547f7f44f-h92tc"]
Sep 15 19:21:18.816012 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.815989    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"]
Sep 15 19:21:18.817469 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:18.817451    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-16a967-6959bfb6b9-kn8vr"]
Sep 15 19:21:20.100069 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:20.100039    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12c2ed7d-e638-4f27-b6b6-aa61ab876b42" path="/var/lib/kubelet/pods/12c2ed7d-e638-4f27-b6b6-aa61ab876b42/volumes"
Sep 15 19:21:20.100453 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:20.100417    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a" path="/var/lib/kubelet/pods/c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a/volumes"
Sep 15 19:21:35.357273 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.357237    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"]
Sep 15 19:21:35.357858 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.357504    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9" podUID="e55df82e-cab7-4254-8331-0280be195c4d" containerName="maas-api" containerID="cri-o://1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f" gracePeriod=30
Sep 15 19:21:35.410419 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.410375    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9" podUID="e55df82e-cab7-4254-8331-0280be195c4d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.44:8443/health\": dial tcp 10.134.0.44:8443: connect: connection refused"
Sep 15 19:21:35.417034 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.417006    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"]
Sep 15 19:21:35.417269 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.417227    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s" podUID="493dfe32-ca79-4d34-9775-116c767bc1d1" containerName="payload-processing" containerID="cri-o://a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1" gracePeriod=30
Sep 15 19:21:35.425204 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.425179    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"]
Sep 15 19:21:35.425838 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.425706    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4" podUID="1f1f00a0-4794-4481-9d71-fb795ae77e91" containerName="payload-pre-processing" containerID="cri-o://71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61" gracePeriod=30
Sep 15 19:21:35.687641 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.687614    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:21:35.712501 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.712478    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:21:35.720219 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.720200    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:21:35.808868 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.808839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493dfe32-ca79-4d34-9775-116c767bc1d1-plugins-config-volume\") pod \"493dfe32-ca79-4d34-9775-116c767bc1d1\" (UID: \"493dfe32-ca79-4d34-9775-116c767bc1d1\") "
Sep 15 19:21:35.809041 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.808909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-maas-api-tls\") pod \"e55df82e-cab7-4254-8331-0280be195c4d\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") "
Sep 15 19:21:35.809041 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.808931    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86sv8\" (UniqueName: \"kubernetes.io/projected/1f1f00a0-4794-4481-9d71-fb795ae77e91-kube-api-access-86sv8\") pod \"1f1f00a0-4794-4481-9d71-fb795ae77e91\" (UID: \"1f1f00a0-4794-4481-9d71-fb795ae77e91\") "
Sep 15 19:21:35.809041 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.808973    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-metrics-certs\") pod \"e55df82e-cab7-4254-8331-0280be195c4d\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") "
Sep 15 19:21:35.809041 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.808987    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h44qk\" (UniqueName: \"kubernetes.io/projected/e55df82e-cab7-4254-8331-0280be195c4d-kube-api-access-h44qk\") pod \"e55df82e-cab7-4254-8331-0280be195c4d\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") "
Sep 15 19:21:35.809041 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.809006    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55df82e-cab7-4254-8331-0280be195c4d-openshift-service-ca\") pod \"e55df82e-cab7-4254-8331-0280be195c4d\" (UID: \"e55df82e-cab7-4254-8331-0280be195c4d\") "
Sep 15 19:21:35.809041 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.809024    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1f00a0-4794-4481-9d71-fb795ae77e91-plugins-config-volume\") pod \"1f1f00a0-4794-4481-9d71-fb795ae77e91\" (UID: \"1f1f00a0-4794-4481-9d71-fb795ae77e91\") "
Sep 15 19:21:35.810219 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.809150    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sfsb\" (UniqueName: \"kubernetes.io/projected/493dfe32-ca79-4d34-9775-116c767bc1d1-kube-api-access-4sfsb\") pod \"493dfe32-ca79-4d34-9775-116c767bc1d1\" (UID: \"493dfe32-ca79-4d34-9775-116c767bc1d1\") "
Sep 15 19:21:35.810219 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.809465    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1f00a0-4794-4481-9d71-fb795ae77e91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f1f00a0-4794-4481-9d71-fb795ae77e91" (UID: "1f1f00a0-4794-4481-9d71-fb795ae77e91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:35.810219 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.809476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493dfe32-ca79-4d34-9775-116c767bc1d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "493dfe32-ca79-4d34-9775-116c767bc1d1" (UID: "493dfe32-ca79-4d34-9775-116c767bc1d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:35.810219 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.809719    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1f00a0-4794-4481-9d71-fb795ae77e91-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:35.810219 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.809741    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493dfe32-ca79-4d34-9775-116c767bc1d1-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:35.810219 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.810166    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55df82e-cab7-4254-8331-0280be195c4d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e55df82e-cab7-4254-8331-0280be195c4d" (UID: "e55df82e-cab7-4254-8331-0280be195c4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:35.811272 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.811244    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e55df82e-cab7-4254-8331-0280be195c4d" (UID: "e55df82e-cab7-4254-8331-0280be195c4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:35.811427 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.811407    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55df82e-cab7-4254-8331-0280be195c4d-kube-api-access-h44qk" (OuterVolumeSpecName: "kube-api-access-h44qk") pod "e55df82e-cab7-4254-8331-0280be195c4d" (UID: "e55df82e-cab7-4254-8331-0280be195c4d"). InnerVolumeSpecName "kube-api-access-h44qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:35.811506 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.811481    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e55df82e-cab7-4254-8331-0280be195c4d" (UID: "e55df82e-cab7-4254-8331-0280be195c4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:35.811715 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.811698    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1f00a0-4794-4481-9d71-fb795ae77e91-kube-api-access-86sv8" (OuterVolumeSpecName: "kube-api-access-86sv8") pod "1f1f00a0-4794-4481-9d71-fb795ae77e91" (UID: "1f1f00a0-4794-4481-9d71-fb795ae77e91"). InnerVolumeSpecName "kube-api-access-86sv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:35.811715 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.811704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493dfe32-ca79-4d34-9775-116c767bc1d1-kube-api-access-4sfsb" (OuterVolumeSpecName: "kube-api-access-4sfsb") pod "493dfe32-ca79-4d34-9775-116c767bc1d1" (UID: "493dfe32-ca79-4d34-9775-116c767bc1d1"). InnerVolumeSpecName "kube-api-access-4sfsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:35.856458 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.856429    2578 generic.go:356] "Generic (PLEG): container finished" podID="1f1f00a0-4794-4481-9d71-fb795ae77e91" containerID="71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61" exitCode=0
Sep 15 19:21:35.856587 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.856482    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"
Sep 15 19:21:35.856587 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.856518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4" event={"ID":"1f1f00a0-4794-4481-9d71-fb795ae77e91","Type":"ContainerDied","Data":"71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61"}
Sep 15 19:21:35.856688 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.856595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4" event={"ID":"1f1f00a0-4794-4481-9d71-fb795ae77e91","Type":"ContainerDied","Data":"2ff33c8cf59a7279e439917823b557ebc9380322c3c89f32436aa97a566b7a2e"}
Sep 15 19:21:35.856688 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.856626    2578 scope.go:117] "RemoveContainer" containerID="71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61"
Sep 15 19:21:35.857965 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.857941    2578 generic.go:356] "Generic (PLEG): container finished" podID="493dfe32-ca79-4d34-9775-116c767bc1d1" containerID="a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1" exitCode=0
Sep 15 19:21:35.858083 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.858000    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"
Sep 15 19:21:35.858083 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.858032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s" event={"ID":"493dfe32-ca79-4d34-9775-116c767bc1d1","Type":"ContainerDied","Data":"a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1"}
Sep 15 19:21:35.858198 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.858084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s" event={"ID":"493dfe32-ca79-4d34-9775-116c767bc1d1","Type":"ContainerDied","Data":"a3cfbb476f5f8ed3bd576c3a1e6c497531788c24aff88439afb5a037d9f6ea12"}
Sep 15 19:21:35.859471 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.859451    2578 generic.go:356] "Generic (PLEG): container finished" podID="e55df82e-cab7-4254-8331-0280be195c4d" containerID="1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f" exitCode=0
Sep 15 19:21:35.859593 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.859487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9" event={"ID":"e55df82e-cab7-4254-8331-0280be195c4d","Type":"ContainerDied","Data":"1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f"}
Sep 15 19:21:35.859593 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.859496    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"
Sep 15 19:21:35.859593 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.859529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9" event={"ID":"e55df82e-cab7-4254-8331-0280be195c4d","Type":"ContainerDied","Data":"b16ba718ba1b1535c8aa2d0a1bb8869acd002bec203e7cd22533e294eae73ebe"}
Sep 15 19:21:35.865296 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.865276    2578 scope.go:117] "RemoveContainer" containerID="71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61"
Sep 15 19:21:35.865573 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:21:35.865554    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61\": container with ID starting with 71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61 not found: ID does not exist" containerID="71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61"
Sep 15 19:21:35.865642 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.865582    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61"} err="failed to get container status \"71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61\": rpc error: code = NotFound desc = could not find container \"71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61\": container with ID starting with 71b99189b774fc2ca6264ad278610e51460efd258ded5d01ec08697efc308f61 not found: ID does not exist"
Sep 15 19:21:35.865642 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.865599    2578 scope.go:117] "RemoveContainer" containerID="a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1"
Sep 15 19:21:35.874196 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.874181    2578 scope.go:117] "RemoveContainer" containerID="a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1"
Sep 15 19:21:35.874465 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:21:35.874447    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1\": container with ID starting with a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1 not found: ID does not exist" containerID="a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1"
Sep 15 19:21:35.874510 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.874473    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1"} err="failed to get container status \"a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1\": rpc error: code = NotFound desc = could not find container \"a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1\": container with ID starting with a2125158769816221b568d56c13179e62fb7d8d9db5a17d965dd0dda15bc79b1 not found: ID does not exist"
Sep 15 19:21:35.874510 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.874489    2578 scope.go:117] "RemoveContainer" containerID="1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f"
Sep 15 19:21:35.882188 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.882169    2578 scope.go:117] "RemoveContainer" containerID="1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f"
Sep 15 19:21:35.882454 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:21:35.882438    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f\": container with ID starting with 1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f not found: ID does not exist" containerID="1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f"
Sep 15 19:21:35.882507 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.882460    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f"} err="failed to get container status \"1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f\": rpc error: code = NotFound desc = could not find container \"1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f\": container with ID starting with 1849aca6701cda29b1c60ba896836108790ffc5001c4d7bba618ab06acf2e94f not found: ID does not exist"
Sep 15 19:21:35.883464 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.883444    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"]
Sep 15 19:21:35.887924 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.887902    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-362ea4-6f4c6b5d6-65vc4"]
Sep 15 19:21:35.897170 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.897150    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"]
Sep 15 19:21:35.900517 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.900498    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-362ea4-6f7c59945c-t7t5s"]
Sep 15 19:21:35.909722 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.909702    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"]
Sep 15 19:21:35.910336 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.910285    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:35.910423 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.910336    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86sv8\" (UniqueName: \"kubernetes.io/projected/1f1f00a0-4794-4481-9d71-fb795ae77e91-kube-api-access-86sv8\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:35.910423 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.910351    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55df82e-cab7-4254-8331-0280be195c4d-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:35.910423 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.910366    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h44qk\" (UniqueName: \"kubernetes.io/projected/e55df82e-cab7-4254-8331-0280be195c4d-kube-api-access-h44qk\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:35.910423 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.910379    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55df82e-cab7-4254-8331-0280be195c4d-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:35.910423 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.910392    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfsb\" (UniqueName: \"kubernetes.io/projected/493dfe32-ca79-4d34-9775-116c767bc1d1-kube-api-access-4sfsb\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:35.911695 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:35.911678    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-362ea4-55dcf766bf-2k8l9"]
Sep 15 19:21:36.088687 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:36.088654    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:21:36.098476 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:36.098448    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f1f00a0-4794-4481-9d71-fb795ae77e91" path="/var/lib/kubelet/pods/1f1f00a0-4794-4481-9d71-fb795ae77e91/volumes"
Sep 15 19:21:36.098815 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:36.098787    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493dfe32-ca79-4d34-9775-116c767bc1d1" path="/var/lib/kubelet/pods/493dfe32-ca79-4d34-9775-116c767bc1d1/volumes"
Sep 15 19:21:36.099155 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:36.099143    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e55df82e-cab7-4254-8331-0280be195c4d" path="/var/lib/kubelet/pods/e55df82e-cab7-4254-8331-0280be195c4d/volumes"
Sep 15 19:21:46.981476 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:46.981437    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:21:53.016316 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.016281    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"]
Sep 15 19:21:53.016815 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.016533    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp" podUID="50f9e362-7091-4a00-bb76-f651a1588bc7" containerName="payload-pre-processing" containerID="cri-o://39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7" gracePeriod=30
Sep 15 19:21:53.272458 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.272378    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:21:53.453741 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.453702    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7992\" (UniqueName: \"kubernetes.io/projected/50f9e362-7091-4a00-bb76-f651a1588bc7-kube-api-access-k7992\") pod \"50f9e362-7091-4a00-bb76-f651a1588bc7\" (UID: \"50f9e362-7091-4a00-bb76-f651a1588bc7\") "
Sep 15 19:21:53.453975 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.453754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f9e362-7091-4a00-bb76-f651a1588bc7-plugins-config-volume\") pod \"50f9e362-7091-4a00-bb76-f651a1588bc7\" (UID: \"50f9e362-7091-4a00-bb76-f651a1588bc7\") "
Sep 15 19:21:53.454224 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.454189    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f9e362-7091-4a00-bb76-f651a1588bc7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50f9e362-7091-4a00-bb76-f651a1588bc7" (UID: "50f9e362-7091-4a00-bb76-f651a1588bc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:53.455927 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.455901    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f9e362-7091-4a00-bb76-f651a1588bc7-kube-api-access-k7992" (OuterVolumeSpecName: "kube-api-access-k7992") pod "50f9e362-7091-4a00-bb76-f651a1588bc7" (UID: "50f9e362-7091-4a00-bb76-f651a1588bc7"). InnerVolumeSpecName "kube-api-access-k7992". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:53.554630 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.554543    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7992\" (UniqueName: \"kubernetes.io/projected/50f9e362-7091-4a00-bb76-f651a1588bc7-kube-api-access-k7992\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:53.554630 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.554575    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f9e362-7091-4a00-bb76-f651a1588bc7-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:21:53.945113 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.945075    2578 generic.go:356] "Generic (PLEG): container finished" podID="50f9e362-7091-4a00-bb76-f651a1588bc7" containerID="39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7" exitCode=0
Sep 15 19:21:53.945281 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.945123    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"
Sep 15 19:21:53.945281 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.945161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp" event={"ID":"50f9e362-7091-4a00-bb76-f651a1588bc7","Type":"ContainerDied","Data":"39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7"}
Sep 15 19:21:53.945281 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.945238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp" event={"ID":"50f9e362-7091-4a00-bb76-f651a1588bc7","Type":"ContainerDied","Data":"e2d6982c4e9af97d34c3beba70dc967c71220116d6e43598f83261f33a74eb27"}
Sep 15 19:21:53.945281 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.945262    2578 scope.go:117] "RemoveContainer" containerID="39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7"
Sep 15 19:21:53.953588 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.953571    2578 scope.go:117] "RemoveContainer" containerID="39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7"
Sep 15 19:21:53.953867 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:21:53.953846    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7\": container with ID starting with 39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7 not found: ID does not exist" containerID="39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7"
Sep 15 19:21:53.953925 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.953875    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7"} err="failed to get container status \"39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7\": rpc error: code = NotFound desc = could not find container \"39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7\": container with ID starting with 39caf1b880035d2af6c2e22d1afe6c4057cdf8adb94c7eafa41be1ad8787f0b7 not found: ID does not exist"
Sep 15 19:21:53.964683 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.964652    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"]
Sep 15 19:21:53.966574 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:53.966554    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1b0e3f-7fd947c8fd-7jjqp"]
Sep 15 19:21:54.098246 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:54.098217    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f9e362-7091-4a00-bb76-f651a1588bc7" path="/var/lib/kubelet/pods/50f9e362-7091-4a00-bb76-f651a1588bc7/volumes"
Sep 15 19:21:57.571107 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:21:57.571074    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:22:09.077289 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:22:09.077252    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:22:32.776370 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:22:32.776289    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:22:47.872937 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:22:47.872897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:22:58.079910 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:22:58.079879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:23:14.988739 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:14.988701    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:23:32.273837 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:32.273787    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:23:42.391058 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:42.391021    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:23:54.717605 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.717564    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"]
Sep 15 19:23:54.718063 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718046    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c2ed7d-e638-4f27-b6b6-aa61ab876b42" containerName="maas-api"
Sep 15 19:23:54.718112 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718064    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c2ed7d-e638-4f27-b6b6-aa61ab876b42" containerName="maas-api"
Sep 15 19:23:54.718112 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718078    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55df82e-cab7-4254-8331-0280be195c4d" containerName="maas-api"
Sep 15 19:23:54.718112 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718083    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55df82e-cab7-4254-8331-0280be195c4d" containerName="maas-api"
Sep 15 19:23:54.718112 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718093    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad6ea98-8625-427e-ad4e-5fb504ff8dd0" containerName="payload-processing"
Sep 15 19:23:54.718112 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718099    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad6ea98-8625-427e-ad4e-5fb504ff8dd0" containerName="payload-processing"
Sep 15 19:23:54.718112 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718104    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f9e362-7091-4a00-bb76-f651a1588bc7" containerName="payload-pre-processing"
Sep 15 19:23:54.718112 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718109    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f9e362-7091-4a00-bb76-f651a1588bc7" containerName="payload-pre-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718116    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b1d9a0-2fc4-445b-afc5-97b084e41947" containerName="maas-api"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718121    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b1d9a0-2fc4-445b-afc5-97b084e41947" containerName="maas-api"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718129    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7954e7c-224c-4f41-aed2-d68b31550af8" containerName="payload-pre-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718134    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7954e7c-224c-4f41-aed2-d68b31550af8" containerName="payload-pre-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718142    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f1f00a0-4794-4481-9d71-fb795ae77e91" containerName="payload-pre-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718147    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f1f00a0-4794-4481-9d71-fb795ae77e91" containerName="payload-pre-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718158    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a" containerName="payload-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718163    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a" containerName="payload-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718170    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33fda78e-3d3b-4288-8368-f7e7d0c518f2" containerName="main"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718174    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33fda78e-3d3b-4288-8368-f7e7d0c518f2" containerName="main"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718182    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493dfe32-ca79-4d34-9775-116c767bc1d1" containerName="payload-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718188    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="493dfe32-ca79-4d34-9775-116c767bc1d1" containerName="payload-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718246    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c2ed7d-e638-4f27-b6b6-aa61ab876b42" containerName="maas-api"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718253    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f9e362-7091-4a00-bb76-f651a1588bc7" containerName="payload-pre-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718261    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f1f00a0-4794-4481-9d71-fb795ae77e91" containerName="payload-pre-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718267    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad6ea98-8625-427e-ad4e-5fb504ff8dd0" containerName="payload-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718274    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e55df82e-cab7-4254-8331-0280be195c4d" containerName="maas-api"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718279    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7954e7c-224c-4f41-aed2-d68b31550af8" containerName="payload-pre-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718284    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="33fda78e-3d3b-4288-8368-f7e7d0c518f2" containerName="main"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718289    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b1d9a0-2fc4-445b-afc5-97b084e41947" containerName="maas-api"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718294    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8aedf5a-46fb-4e8a-bd1c-6e0ea6b6179a" containerName="payload-processing"
Sep 15 19:23:54.718333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.718299    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="493dfe32-ca79-4d34-9775-116c767bc1d1" containerName="payload-processing"
Sep 15 19:23:54.721059 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.721045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.723337 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.723313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f61404-serving-cert\""
Sep 15 19:23:54.730903 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.730875    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"]
Sep 15 19:23:54.738451 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.738427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80222f3d-7154-4282-966f-f09402a4c66a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.738560 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.738474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.738560 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.738496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsdt\" (UniqueName: \"kubernetes.io/projected/80222f3d-7154-4282-966f-f09402a4c66a-kube-api-access-jnsdt\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.738652 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.738571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-metrics-certs\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.839315 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.839278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsdt\" (UniqueName: \"kubernetes.io/projected/80222f3d-7154-4282-966f-f09402a4c66a-kube-api-access-jnsdt\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.839315 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.839321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-metrics-certs\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.839562 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.839365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80222f3d-7154-4282-966f-f09402a4c66a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.839562 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.839405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.840121 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.840090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80222f3d-7154-4282-966f-f09402a4c66a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.841763 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.841745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-metrics-certs\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.841837 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.841788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:54.847084 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:54.847059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsdt\" (UniqueName: \"kubernetes.io/projected/80222f3d-7154-4282-966f-f09402a4c66a-kube-api-access-jnsdt\") pod \"maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:55.033049 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.032963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:55.304743 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.304668    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"]
Sep 15 19:23:55.307741 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:23:55.307707    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80222f3d_7154_4282_966f_f09402a4c66a.slice/crio-eca73501e973a5a3ac6959987cc9b0301f479ef7534a8e229b9c8ebcd6b4cd01 WatchSource:0}: Error finding container eca73501e973a5a3ac6959987cc9b0301f479ef7534a8e229b9c8ebcd6b4cd01: Status 404 returned error can't find the container with id eca73501e973a5a3ac6959987cc9b0301f479ef7534a8e229b9c8ebcd6b4cd01
Sep 15 19:23:55.402872 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.402835    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"]
Sep 15 19:23:55.407080 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.407062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:23:55.409730 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.409706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f61404-dockercfg-jrhqn\""
Sep 15 19:23:55.409730 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.409707    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f61404\""
Sep 15 19:23:55.416901 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.416875    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"]
Sep 15 19:23:55.444194 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.444160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm\" (UID: \"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:23:55.444349 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.444209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgd5\" (UniqueName: \"kubernetes.io/projected/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-kube-api-access-5jgd5\") pod \"payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm\" (UID: \"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:23:55.475588 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.475555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb" event={"ID":"80222f3d-7154-4282-966f-f09402a4c66a","Type":"ContainerStarted","Data":"eca73501e973a5a3ac6959987cc9b0301f479ef7534a8e229b9c8ebcd6b4cd01"}
Sep 15 19:23:55.544983 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.544945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgd5\" (UniqueName: \"kubernetes.io/projected/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-kube-api-access-5jgd5\") pod \"payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm\" (UID: \"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:23:55.545164 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.545043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm\" (UID: \"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:23:55.545702 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.545683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm\" (UID: \"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:23:55.552132 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.552105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgd5\" (UniqueName: \"kubernetes.io/projected/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-kube-api-access-5jgd5\") pod \"payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm\" (UID: \"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:23:55.718347 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.718308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:23:55.991397 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:55.991324    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"]
Sep 15 19:23:55.994368 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:23:55.994340    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b8d06e_be59_4bbe_a6f1_ba443b3a4668.slice/crio-94560deed90e1e55c6aa74e15fc857b5403f67a44627a2a7705421328229a572 WatchSource:0}: Error finding container 94560deed90e1e55c6aa74e15fc857b5403f67a44627a2a7705421328229a572: Status 404 returned error can't find the container with id 94560deed90e1e55c6aa74e15fc857b5403f67a44627a2a7705421328229a572
Sep 15 19:23:56.482243 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:56.482204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm" event={"ID":"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668","Type":"ContainerStarted","Data":"69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec"}
Sep 15 19:23:56.482451 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:56.482267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm" event={"ID":"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668","Type":"ContainerStarted","Data":"94560deed90e1e55c6aa74e15fc857b5403f67a44627a2a7705421328229a572"}
Sep 15 19:23:56.482451 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:56.482292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:23:56.483882 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:56.483849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb" event={"ID":"80222f3d-7154-4282-966f-f09402a4c66a","Type":"ContainerStarted","Data":"0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde"}
Sep 15 19:23:56.484019 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:56.483972    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:23:56.501220 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:56.501170    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm" podStartSLOduration=1.5011558900000002 podStartE2EDuration="1.50115589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:23:56.499341463 +0000 UTC m=+1987.041436993" watchObservedRunningTime="2026-09-15 19:23:56.50115589 +0000 UTC m=+1987.043251464"
Sep 15 19:23:56.515744 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:56.515694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb" podStartSLOduration=2.101535071 podStartE2EDuration="2.515680538s" totalImagesPullingTime="414.145467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:23:56.514654473 +0000 UTC m=+1987.056750007" watchObservedRunningTime="2026-09-15 19:23:56.515680538 +0000 UTC m=+1987.057776066"
Sep 15 19:23:58.074634 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:23:58.074596    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:24:02.493831 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:02.493788    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:24:07.490849 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:07.490818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:24:24.445504 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.445464    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"]
Sep 15 19:24:24.449113 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.449094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.451176 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.451157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0ce162-serving-cert\""
Sep 15 19:24:24.458039 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.458005    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"]
Sep 15 19:24:24.571473 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.571434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c3ff10-076e-4c15-92cf-0843a7fcb89e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.571642 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.571482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.571642 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.571506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.571642 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.571534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwxm5\" (UniqueName: \"kubernetes.io/projected/24c3ff10-076e-4c15-92cf-0843a7fcb89e-kube-api-access-dwxm5\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.672947 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.672907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c3ff10-076e-4c15-92cf-0843a7fcb89e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.673125 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.672962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.673125 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.672991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.673125 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.673013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwxm5\" (UniqueName: \"kubernetes.io/projected/24c3ff10-076e-4c15-92cf-0843a7fcb89e-kube-api-access-dwxm5\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.673230 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:24:24.673144    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-serving-cert: secret "maas-api-e2e-ipp-b-0ce162-serving-cert" not found
Sep 15 19:24:24.673269 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:24:24.673232    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls podName:24c3ff10-076e-4c15-92cf-0843a7fcb89e nodeName:}" failed. No retries permitted until 2026-09-15 19:24:25.173209587 +0000 UTC m=+2015.715305095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls") pod "maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856" (UID: "24c3ff10-076e-4c15-92cf-0843a7fcb89e") : secret "maas-api-e2e-ipp-b-0ce162-serving-cert" not found
Sep 15 19:24:24.673649 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.673629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c3ff10-076e-4c15-92cf-0843a7fcb89e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.675536 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.675513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:24.685379 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:24.681995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwxm5\" (UniqueName: \"kubernetes.io/projected/24c3ff10-076e-4c15-92cf-0843a7fcb89e-kube-api-access-dwxm5\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:25.177036 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:25.176994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:25.177321 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:24:25.177292    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-serving-cert: secret "maas-api-e2e-ipp-b-0ce162-serving-cert" not found
Sep 15 19:24:25.177406 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:24:25.177388    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls podName:24c3ff10-076e-4c15-92cf-0843a7fcb89e nodeName:}" failed. No retries permitted until 2026-09-15 19:24:26.177364387 +0000 UTC m=+2016.719459895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls") pod "maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856" (UID: "24c3ff10-076e-4c15-92cf-0843a7fcb89e") : secret "maas-api-e2e-ipp-b-0ce162-serving-cert" not found
Sep 15 19:24:26.187189 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:26.187150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:26.189539 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:26.189504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:26.261284 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:26.261243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:26.533369 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:26.533342    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"]
Sep 15 19:24:26.538098 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:24:26.538063    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c3ff10_076e_4c15_92cf_0843a7fcb89e.slice/crio-993ee7ba5e2847b340bef3046b588f098a96ea3200e7b5ab1715375f5de0722c WatchSource:0}: Error finding container 993ee7ba5e2847b340bef3046b588f098a96ea3200e7b5ab1715375f5de0722c: Status 404 returned error can't find the container with id 993ee7ba5e2847b340bef3046b588f098a96ea3200e7b5ab1715375f5de0722c
Sep 15 19:24:26.616953 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:26.616912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856" event={"ID":"24c3ff10-076e-4c15-92cf-0843a7fcb89e","Type":"ContainerStarted","Data":"993ee7ba5e2847b340bef3046b588f098a96ea3200e7b5ab1715375f5de0722c"}
Sep 15 19:24:27.623627 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:27.623583    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856" event={"ID":"24c3ff10-076e-4c15-92cf-0843a7fcb89e","Type":"ContainerStarted","Data":"198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10"}
Sep 15 19:24:27.624037 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:27.623651    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:24:27.640881 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:27.640831    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856" podStartSLOduration=3.108605348 podStartE2EDuration="3.640815027s" totalImagesPullingTime="532.209679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:24:27.640529953 +0000 UTC m=+2018.182625487" watchObservedRunningTime="2026-09-15 19:24:27.640815027 +0000 UTC m=+2018.182910545"
Sep 15 19:24:29.893025 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:29.892984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:24:33.634599 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:24:33.634563    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:25:12.687454 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:12.687413    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:25:32.517026 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:32.516992    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:25:38.779187 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:38.779155    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm_c4b8d06e-be59-4bbe-a6f1-ba443b3a4668/payload-processing/0.log"
Sep 15 19:25:46.497328 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.497295    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"]
Sep 15 19:25:46.500742 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.500721    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.503447 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.503423    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-f8b852-serving-cert\""
Sep 15 19:25:46.510595 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.510566    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"]
Sep 15 19:25:46.576504 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.576464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.576709 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.576536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.576709 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.576569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t5r6\" (UniqueName: \"kubernetes.io/projected/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-kube-api-access-9t5r6\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.576709 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.576626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.677566 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.677517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.677566 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.677571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.677882 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.677683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t5r6\" (UniqueName: \"kubernetes.io/projected/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-kube-api-access-9t5r6\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.677882 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.677750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.677882 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:25:46.677871    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-serving-cert: secret "maas-api-e2e-ipp-cleanup-f8b852-serving-cert" not found
Sep 15 19:25:46.678061 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:25:46.677955    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-maas-api-tls podName:d81d4ecc-b9d7-403c-9af8-c396edd06ce8 nodeName:}" failed. No retries permitted until 2026-09-15 19:25:47.177930669 +0000 UTC m=+2097.720026181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2" (UID: "d81d4ecc-b9d7-403c-9af8-c396edd06ce8") : secret "maas-api-e2e-ipp-cleanup-f8b852-serving-cert" not found
Sep 15 19:25:46.678302 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.678279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.680221 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.680199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:46.685874 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:46.685858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t5r6\" (UniqueName: \"kubernetes.io/projected/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-kube-api-access-9t5r6\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:47.183083 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:47.183047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:47.185443 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:47.185425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:47.411681 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:47.411649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:47.682856 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:47.682830    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"]
Sep 15 19:25:47.685110 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:25:47.685074    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81d4ecc_b9d7_403c_9af8_c396edd06ce8.slice/crio-a93ded8e1e0b7a56c9ab9b32ef74882dfea0824136773968cb0083602e86cf1f WatchSource:0}: Error finding container a93ded8e1e0b7a56c9ab9b32ef74882dfea0824136773968cb0083602e86cf1f: Status 404 returned error can't find the container with id a93ded8e1e0b7a56c9ab9b32ef74882dfea0824136773968cb0083602e86cf1f
Sep 15 19:25:47.686429 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:47.686411    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:25:47.981766 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:47.981718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2" event={"ID":"d81d4ecc-b9d7-403c-9af8-c396edd06ce8","Type":"ContainerStarted","Data":"a93ded8e1e0b7a56c9ab9b32ef74882dfea0824136773968cb0083602e86cf1f"}
Sep 15 19:25:48.988237 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:48.988201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2" event={"ID":"d81d4ecc-b9d7-403c-9af8-c396edd06ce8","Type":"ContainerStarted","Data":"7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf"}
Sep 15 19:25:48.988647 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:48.988281    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:25:49.020685 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:49.020635    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2" podStartSLOduration=2.6058552329999998 podStartE2EDuration="3.020619432s" totalImagesPullingTime="414.764199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:25:49.019563578 +0000 UTC m=+2099.561659108" watchObservedRunningTime="2026-09-15 19:25:49.020619432 +0000 UTC m=+2099.562714961"
Sep 15 19:25:51.087511 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:51.087474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:25:54.998645 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:25:54.998615    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:26:25.060685 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.060651    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"]
Sep 15 19:26:25.063350 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.061156    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2" podUID="d81d4ecc-b9d7-403c-9af8-c396edd06ce8" containerName="maas-api" containerID="cri-o://7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf" gracePeriod=30
Sep 15 19:26:25.315279 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.315214    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:26:25.423414 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.423382    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-openshift-service-ca\") pod \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") "
Sep 15 19:26:25.423414 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.423415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-metrics-certs\") pod \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") "
Sep 15 19:26:25.423650 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.423462    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-maas-api-tls\") pod \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") "
Sep 15 19:26:25.423650 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.423537    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t5r6\" (UniqueName: \"kubernetes.io/projected/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-kube-api-access-9t5r6\") pod \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\" (UID: \"d81d4ecc-b9d7-403c-9af8-c396edd06ce8\") "
Sep 15 19:26:25.423771 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.423749    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d81d4ecc-b9d7-403c-9af8-c396edd06ce8" (UID: "d81d4ecc-b9d7-403c-9af8-c396edd06ce8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:26:25.425590 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.425563    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d81d4ecc-b9d7-403c-9af8-c396edd06ce8" (UID: "d81d4ecc-b9d7-403c-9af8-c396edd06ce8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:26:25.425686 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.425593    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d81d4ecc-b9d7-403c-9af8-c396edd06ce8" (UID: "d81d4ecc-b9d7-403c-9af8-c396edd06ce8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:26:25.425686 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.425645    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-kube-api-access-9t5r6" (OuterVolumeSpecName: "kube-api-access-9t5r6") pod "d81d4ecc-b9d7-403c-9af8-c396edd06ce8" (UID: "d81d4ecc-b9d7-403c-9af8-c396edd06ce8"). InnerVolumeSpecName "kube-api-access-9t5r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:26:25.524063 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.524027    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t5r6\" (UniqueName: \"kubernetes.io/projected/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-kube-api-access-9t5r6\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:26:25.524063 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.524055    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:26:25.524063 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.524069    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:26:25.524288 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:25.524081    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d81d4ecc-b9d7-403c-9af8-c396edd06ce8-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:26:26.155478 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:26.155446    2578 generic.go:356] "Generic (PLEG): container finished" podID="d81d4ecc-b9d7-403c-9af8-c396edd06ce8" containerID="7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf" exitCode=0
Sep 15 19:26:26.155911 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:26.155490    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"
Sep 15 19:26:26.155911 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:26.155529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2" event={"ID":"d81d4ecc-b9d7-403c-9af8-c396edd06ce8","Type":"ContainerDied","Data":"7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf"}
Sep 15 19:26:26.155911 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:26.155579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2" event={"ID":"d81d4ecc-b9d7-403c-9af8-c396edd06ce8","Type":"ContainerDied","Data":"a93ded8e1e0b7a56c9ab9b32ef74882dfea0824136773968cb0083602e86cf1f"}
Sep 15 19:26:26.155911 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:26.155599    2578 scope.go:117] "RemoveContainer" containerID="7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf"
Sep 15 19:26:26.164085 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:26.164057    2578 scope.go:117] "RemoveContainer" containerID="7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf"
Sep 15 19:26:26.164357 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:26:26.164338    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf\": container with ID starting with 7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf not found: ID does not exist" containerID="7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf"
Sep 15 19:26:26.164403 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:26.164365    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf"} err="failed to get container status \"7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf\": rpc error: code = NotFound desc = could not find container \"7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf\": container with ID starting with 7c1b35c904fc7a72157b102c461e2e92a09d234cd76a8248a66e4bbafdbd89bf not found: ID does not exist"
Sep 15 19:26:26.172424 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:26.172400    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"]
Sep 15 19:26:26.178347 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:26.178325    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f8b852-7745fdd867-rdds2"]
Sep 15 19:26:28.098694 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:28.098662    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81d4ecc-b9d7-403c-9af8-c396edd06ce8" path="/var/lib/kubelet/pods/d81d4ecc-b9d7-403c-9af8-c396edd06ce8/volumes"
Sep 15 19:26:29.479620 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:29.479582    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:26:48.384191 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:48.384105    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:26:50.766308 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:50.766274    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"]
Sep 15 19:26:50.766771 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:50.766530    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb" podUID="80222f3d-7154-4282-966f-f09402a4c66a" containerName="maas-api" containerID="cri-o://0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde" gracePeriod=30
Sep 15 19:26:50.821083 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:50.821051    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"]
Sep 15 19:26:50.821295 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:50.821254    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm" podUID="c4b8d06e-be59-4bbe-a6f1-ba443b3a4668" containerName="payload-processing" containerID="cri-o://69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec" gracePeriod=30
Sep 15 19:26:51.065159 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.065134    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:26:51.093933 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.093910    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:26:51.116204 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.116179    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnsdt\" (UniqueName: \"kubernetes.io/projected/80222f3d-7154-4282-966f-f09402a4c66a-kube-api-access-jnsdt\") pod \"80222f3d-7154-4282-966f-f09402a4c66a\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") "
Sep 15 19:26:51.116327 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.116211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-plugins-config-volume\") pod \"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668\" (UID: \"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668\") "
Sep 15 19:26:51.116327 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.116234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80222f3d-7154-4282-966f-f09402a4c66a-openshift-service-ca\") pod \"80222f3d-7154-4282-966f-f09402a4c66a\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") "
Sep 15 19:26:51.116327 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.116310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-maas-api-tls\") pod \"80222f3d-7154-4282-966f-f09402a4c66a\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") "
Sep 15 19:26:51.116506 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.116349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-metrics-certs\") pod \"80222f3d-7154-4282-966f-f09402a4c66a\" (UID: \"80222f3d-7154-4282-966f-f09402a4c66a\") "
Sep 15 19:26:51.116506 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.116397    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jgd5\" (UniqueName: \"kubernetes.io/projected/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-kube-api-access-5jgd5\") pod \"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668\" (UID: \"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668\") "
Sep 15 19:26:51.116615 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.116593    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80222f3d-7154-4282-966f-f09402a4c66a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "80222f3d-7154-4282-966f-f09402a4c66a" (UID: "80222f3d-7154-4282-966f-f09402a4c66a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:26:51.116697 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.116675    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80222f3d-7154-4282-966f-f09402a4c66a-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:26:51.117055 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.117020    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4b8d06e-be59-4bbe-a6f1-ba443b3a4668" (UID: "c4b8d06e-be59-4bbe-a6f1-ba443b3a4668"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:26:51.118412 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.118392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "80222f3d-7154-4282-966f-f09402a4c66a" (UID: "80222f3d-7154-4282-966f-f09402a4c66a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:26:51.118599 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.118575    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-kube-api-access-5jgd5" (OuterVolumeSpecName: "kube-api-access-5jgd5") pod "c4b8d06e-be59-4bbe-a6f1-ba443b3a4668" (UID: "c4b8d06e-be59-4bbe-a6f1-ba443b3a4668"). InnerVolumeSpecName "kube-api-access-5jgd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:26:51.118684 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.118609    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80222f3d-7154-4282-966f-f09402a4c66a" (UID: "80222f3d-7154-4282-966f-f09402a4c66a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:26:51.119206 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.119182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80222f3d-7154-4282-966f-f09402a4c66a-kube-api-access-jnsdt" (OuterVolumeSpecName: "kube-api-access-jnsdt") pod "80222f3d-7154-4282-966f-f09402a4c66a" (UID: "80222f3d-7154-4282-966f-f09402a4c66a"). InnerVolumeSpecName "kube-api-access-jnsdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:26:51.217993 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.217965    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgd5\" (UniqueName: \"kubernetes.io/projected/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-kube-api-access-5jgd5\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:26:51.217993 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.217990    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsdt\" (UniqueName: \"kubernetes.io/projected/80222f3d-7154-4282-966f-f09402a4c66a-kube-api-access-jnsdt\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:26:51.218178 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.218000    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:26:51.218178 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.218010    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:26:51.218178 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.218025    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80222f3d-7154-4282-966f-f09402a4c66a-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:26:51.266735 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.266700    2578 generic.go:356] "Generic (PLEG): container finished" podID="c4b8d06e-be59-4bbe-a6f1-ba443b3a4668" containerID="69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec" exitCode=0
Sep 15 19:26:51.266913 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.266745    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"
Sep 15 19:26:51.266913 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.266779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm" event={"ID":"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668","Type":"ContainerDied","Data":"69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec"}
Sep 15 19:26:51.266913 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.266847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm" event={"ID":"c4b8d06e-be59-4bbe-a6f1-ba443b3a4668","Type":"ContainerDied","Data":"94560deed90e1e55c6aa74e15fc857b5403f67a44627a2a7705421328229a572"}
Sep 15 19:26:51.266913 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.266872    2578 scope.go:117] "RemoveContainer" containerID="69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec"
Sep 15 19:26:51.268336 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.268314    2578 generic.go:356] "Generic (PLEG): container finished" podID="80222f3d-7154-4282-966f-f09402a4c66a" containerID="0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde" exitCode=0
Sep 15 19:26:51.268465 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.268376    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"
Sep 15 19:26:51.268465 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.268388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb" event={"ID":"80222f3d-7154-4282-966f-f09402a4c66a","Type":"ContainerDied","Data":"0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde"}
Sep 15 19:26:51.268465 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.268442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb" event={"ID":"80222f3d-7154-4282-966f-f09402a4c66a","Type":"ContainerDied","Data":"eca73501e973a5a3ac6959987cc9b0301f479ef7534a8e229b9c8ebcd6b4cd01"}
Sep 15 19:26:51.275572 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.275553    2578 scope.go:117] "RemoveContainer" containerID="69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec"
Sep 15 19:26:51.275885 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:26:51.275866    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec\": container with ID starting with 69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec not found: ID does not exist" containerID="69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec"
Sep 15 19:26:51.275965 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.275892    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec"} err="failed to get container status \"69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec\": rpc error: code = NotFound desc = could not find container \"69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec\": container with ID starting with 69d0d4c3f2a4de1ad1475355aa213ea2f693d8768e41ff96d249d316058603ec not found: ID does not exist"
Sep 15 19:26:51.275965 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.275907    2578 scope.go:117] "RemoveContainer" containerID="0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde"
Sep 15 19:26:51.283775 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.283759    2578 scope.go:117] "RemoveContainer" containerID="0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde"
Sep 15 19:26:51.284041 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:26:51.284024    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde\": container with ID starting with 0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde not found: ID does not exist" containerID="0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde"
Sep 15 19:26:51.284091 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.284047    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde"} err="failed to get container status \"0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde\": rpc error: code = NotFound desc = could not find container \"0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde\": container with ID starting with 0c2503cfcedc72a77da7b48bc1d68c8cce7624163563a706732c2fa5cff17bde not found: ID does not exist"
Sep 15 19:26:51.290771 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.290750    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"]
Sep 15 19:26:51.294580 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.294560    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f61404-67fb574c67-s2hzm"]
Sep 15 19:26:51.305921 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.305902    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"]
Sep 15 19:26:51.309516 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:51.309496    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f61404-847cdcddc4-ncvsb"]
Sep 15 19:26:52.099197 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:52.099116    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80222f3d-7154-4282-966f-f09402a4c66a" path="/var/lib/kubelet/pods/80222f3d-7154-4282-966f-f09402a4c66a/volumes"
Sep 15 19:26:52.099566 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:26:52.099453    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4b8d06e-be59-4bbe-a6f1-ba443b3a4668" path="/var/lib/kubelet/pods/c4b8d06e-be59-4bbe-a6f1-ba443b3a4668/volumes"
Sep 15 19:27:01.974847 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:01.974811    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:27:49.910695 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:49.910662    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"]
Sep 15 19:27:49.911343 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:49.910874    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856" podUID="24c3ff10-076e-4c15-92cf-0843a7fcb89e" containerName="maas-api" containerID="cri-o://198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10" gracePeriod=30
Sep 15 19:27:50.153325 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.153297    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:27:50.192462 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.192394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls\") pod \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") "
Sep 15 19:27:50.192462 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.192428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c3ff10-076e-4c15-92cf-0843a7fcb89e-openshift-service-ca\") pod \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") "
Sep 15 19:27:50.192462 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.192447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-metrics-certs\") pod \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") "
Sep 15 19:27:50.192711 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.192496    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwxm5\" (UniqueName: \"kubernetes.io/projected/24c3ff10-076e-4c15-92cf-0843a7fcb89e-kube-api-access-dwxm5\") pod \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\" (UID: \"24c3ff10-076e-4c15-92cf-0843a7fcb89e\") "
Sep 15 19:27:50.192863 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.192828    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c3ff10-076e-4c15-92cf-0843a7fcb89e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24c3ff10-076e-4c15-92cf-0843a7fcb89e" (UID: "24c3ff10-076e-4c15-92cf-0843a7fcb89e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:27:50.194625 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.194594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24c3ff10-076e-4c15-92cf-0843a7fcb89e" (UID: "24c3ff10-076e-4c15-92cf-0843a7fcb89e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:27:50.194711 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.194663    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c3ff10-076e-4c15-92cf-0843a7fcb89e-kube-api-access-dwxm5" (OuterVolumeSpecName: "kube-api-access-dwxm5") pod "24c3ff10-076e-4c15-92cf-0843a7fcb89e" (UID: "24c3ff10-076e-4c15-92cf-0843a7fcb89e"). InnerVolumeSpecName "kube-api-access-dwxm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:27:50.194752 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.194722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24c3ff10-076e-4c15-92cf-0843a7fcb89e" (UID: "24c3ff10-076e-4c15-92cf-0843a7fcb89e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:27:50.293910 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.293876    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.294037 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.293913    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c3ff10-076e-4c15-92cf-0843a7fcb89e-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.294037 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.293929    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c3ff10-076e-4c15-92cf-0843a7fcb89e-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.294037 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.293942    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwxm5\" (UniqueName: \"kubernetes.io/projected/24c3ff10-076e-4c15-92cf-0843a7fcb89e-kube-api-access-dwxm5\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:27:50.531621 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.531543    2578 generic.go:356] "Generic (PLEG): container finished" podID="24c3ff10-076e-4c15-92cf-0843a7fcb89e" containerID="198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10" exitCode=0
Sep 15 19:27:50.531621 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.531586    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"
Sep 15 19:27:50.531823 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.531630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856" event={"ID":"24c3ff10-076e-4c15-92cf-0843a7fcb89e","Type":"ContainerDied","Data":"198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10"}
Sep 15 19:27:50.531823 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.531682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856" event={"ID":"24c3ff10-076e-4c15-92cf-0843a7fcb89e","Type":"ContainerDied","Data":"993ee7ba5e2847b340bef3046b588f098a96ea3200e7b5ab1715375f5de0722c"}
Sep 15 19:27:50.531823 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.531706    2578 scope.go:117] "RemoveContainer" containerID="198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10"
Sep 15 19:27:50.540788 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.540770    2578 scope.go:117] "RemoveContainer" containerID="198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10"
Sep 15 19:27:50.541041 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:27:50.541024    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10\": container with ID starting with 198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10 not found: ID does not exist" containerID="198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10"
Sep 15 19:27:50.541082 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.541051    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10"} err="failed to get container status \"198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10\": rpc error: code = NotFound desc = could not find container \"198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10\": container with ID starting with 198e1335982b4d1da09cece09e09703bbad5dcfbcdfcb065fcee8de07f99ac10 not found: ID does not exist"
Sep 15 19:27:50.551827 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.551808    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"]
Sep 15 19:27:50.553948 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:50.553928    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ce162-66d7bf9c69-gl856"]
Sep 15 19:27:52.098626 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:52.098588    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c3ff10-076e-4c15-92cf-0843a7fcb89e" path="/var/lib/kubelet/pods/24c3ff10-076e-4c15-92cf-0843a7fcb89e/volumes"
Sep 15 19:27:53.171126 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:27:53.171090    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:28:07.602307 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:07.602271    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"]
Sep 15 19:28:07.602882 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:07.602522    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df" podUID="c12e64d5-57a8-4045-9bd3-f840ef58a075" containerName="maas-api" containerID="cri-o://14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d" gracePeriod=30
Sep 15 19:28:07.662897 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:07.662861    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"]
Sep 15 19:28:07.663165 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:07.663104    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx" podUID="ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa" containerName="payload-pre-processing" containerID="cri-o://cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced" gracePeriod=30
Sep 15 19:28:07.908386 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:07.908365    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:28:07.910524 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:07.910500    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:28:08.023859 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.023834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c12e64d5-57a8-4045-9bd3-f840ef58a075-openshift-service-ca\") pod \"c12e64d5-57a8-4045-9bd3-f840ef58a075\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") "
Sep 15 19:28:08.024022 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.023877    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-plugins-config-volume\") pod \"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa\" (UID: \"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa\") "
Sep 15 19:28:08.024022 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.023915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzjt8\" (UniqueName: \"kubernetes.io/projected/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-kube-api-access-fzjt8\") pod \"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa\" (UID: \"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa\") "
Sep 15 19:28:08.024022 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.023941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-metrics-certs\") pod \"c12e64d5-57a8-4045-9bd3-f840ef58a075\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") "
Sep 15 19:28:08.024213 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.024034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-maas-api-tls\") pod \"c12e64d5-57a8-4045-9bd3-f840ef58a075\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") "
Sep 15 19:28:08.024213 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.024076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqtfs\" (UniqueName: \"kubernetes.io/projected/c12e64d5-57a8-4045-9bd3-f840ef58a075-kube-api-access-wqtfs\") pod \"c12e64d5-57a8-4045-9bd3-f840ef58a075\" (UID: \"c12e64d5-57a8-4045-9bd3-f840ef58a075\") "
Sep 15 19:28:08.024318 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.024239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa" (UID: "ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:08.024318 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.024267    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12e64d5-57a8-4045-9bd3-f840ef58a075-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c12e64d5-57a8-4045-9bd3-f840ef58a075" (UID: "c12e64d5-57a8-4045-9bd3-f840ef58a075"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:08.024427 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.024383    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c12e64d5-57a8-4045-9bd3-f840ef58a075-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:08.024427 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.024403    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:08.026040 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.026014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-kube-api-access-fzjt8" (OuterVolumeSpecName: "kube-api-access-fzjt8") pod "ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa" (UID: "ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa"). InnerVolumeSpecName "kube-api-access-fzjt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:08.026177 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.026156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12e64d5-57a8-4045-9bd3-f840ef58a075-kube-api-access-wqtfs" (OuterVolumeSpecName: "kube-api-access-wqtfs") pod "c12e64d5-57a8-4045-9bd3-f840ef58a075" (UID: "c12e64d5-57a8-4045-9bd3-f840ef58a075"). InnerVolumeSpecName "kube-api-access-wqtfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:08.026458 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.026442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c12e64d5-57a8-4045-9bd3-f840ef58a075" (UID: "c12e64d5-57a8-4045-9bd3-f840ef58a075"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:28:08.026512 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.026461    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c12e64d5-57a8-4045-9bd3-f840ef58a075" (UID: "c12e64d5-57a8-4045-9bd3-f840ef58a075"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:28:08.125222 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.125200    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzjt8\" (UniqueName: \"kubernetes.io/projected/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa-kube-api-access-fzjt8\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:08.125222 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.125221    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:08.125343 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.125230    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c12e64d5-57a8-4045-9bd3-f840ef58a075-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:08.125343 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.125238    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqtfs\" (UniqueName: \"kubernetes.io/projected/c12e64d5-57a8-4045-9bd3-f840ef58a075-kube-api-access-wqtfs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:08.615020 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.614990    2578 generic.go:356] "Generic (PLEG): container finished" podID="c12e64d5-57a8-4045-9bd3-f840ef58a075" containerID="14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d" exitCode=0
Sep 15 19:28:08.615468 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.615034    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"
Sep 15 19:28:08.615468 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.615086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df" event={"ID":"c12e64d5-57a8-4045-9bd3-f840ef58a075","Type":"ContainerDied","Data":"14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d"}
Sep 15 19:28:08.615468 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.615149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df" event={"ID":"c12e64d5-57a8-4045-9bd3-f840ef58a075","Type":"ContainerDied","Data":"5a670e56623787b4f40387aad84f8e5042945829fd24cd3f58e68e896276cf84"}
Sep 15 19:28:08.615468 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.615181    2578 scope.go:117] "RemoveContainer" containerID="14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d"
Sep 15 19:28:08.616856 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.616503    2578 generic.go:356] "Generic (PLEG): container finished" podID="ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa" containerID="cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced" exitCode=0
Sep 15 19:28:08.616856 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.616533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx" event={"ID":"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa","Type":"ContainerDied","Data":"cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced"}
Sep 15 19:28:08.616856 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.616561    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"
Sep 15 19:28:08.616856 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.616581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx" event={"ID":"ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa","Type":"ContainerDied","Data":"ea15ef49dcef577ea3614300e1ac0828e7590f8f07965eed357536391d0beed1"}
Sep 15 19:28:08.627135 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.627119    2578 scope.go:117] "RemoveContainer" containerID="14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d"
Sep 15 19:28:08.627462 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:28:08.627442    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d\": container with ID starting with 14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d not found: ID does not exist" containerID="14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d"
Sep 15 19:28:08.627517 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.627471    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d"} err="failed to get container status \"14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d\": rpc error: code = NotFound desc = could not find container \"14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d\": container with ID starting with 14f3c4d9aa068d61d48ec28c0cbabbd0d6193da9755a5ab72cc519fb0117c18d not found: ID does not exist"
Sep 15 19:28:08.627517 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.627487    2578 scope.go:117] "RemoveContainer" containerID="cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced"
Sep 15 19:28:08.634956 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.634939    2578 scope.go:117] "RemoveContainer" containerID="cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced"
Sep 15 19:28:08.635258 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:28:08.635240    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced\": container with ID starting with cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced not found: ID does not exist" containerID="cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced"
Sep 15 19:28:08.635321 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.635266    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced"} err="failed to get container status \"cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced\": rpc error: code = NotFound desc = could not find container \"cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced\": container with ID starting with cdab38303e4dff367b6583a55dc501cd01db3599f443159fb4ce65239e5e4ced not found: ID does not exist"
Sep 15 19:28:08.636118 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.636098    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"]
Sep 15 19:28:08.639874 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.639787    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6d10a-7986d8877b-z82df"]
Sep 15 19:28:08.648632 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.648614    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"]
Sep 15 19:28:08.652151 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:08.652132    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6d10a-5d744f84f8-96rcx"]
Sep 15 19:28:10.103697 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:10.100252    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa" path="/var/lib/kubelet/pods/ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa/volumes"
Sep 15 19:28:10.103697 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:10.100841    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12e64d5-57a8-4045-9bd3-f840ef58a075" path="/var/lib/kubelet/pods/c12e64d5-57a8-4045-9bd3-f840ef58a075/volumes"
Sep 15 19:28:10.883606 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:10.883567    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:28:25.070227 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.070152    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"]
Sep 15 19:28:25.070620 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.070369    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd" podUID="15a6f1f4-2497-4960-9054-7275a473c292" containerName="maas-api" containerID="cri-o://3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d" gracePeriod=30
Sep 15 19:28:25.123855 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.123820    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"]
Sep 15 19:28:25.124076 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.124049    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh" podUID="ea37e1d3-a803-4f98-93d4-589ebbc59e42" containerName="payload-pre-processing" containerID="cri-o://a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd" gracePeriod=30
Sep 15 19:28:25.351363 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.351334    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:28:25.388593 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.388566    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:28:25.455997 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.455968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea37e1d3-a803-4f98-93d4-589ebbc59e42-plugins-config-volume\") pod \"ea37e1d3-a803-4f98-93d4-589ebbc59e42\" (UID: \"ea37e1d3-a803-4f98-93d4-589ebbc59e42\") "
Sep 15 19:28:25.456146 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.456029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g8gh\" (UniqueName: \"kubernetes.io/projected/ea37e1d3-a803-4f98-93d4-589ebbc59e42-kube-api-access-9g8gh\") pod \"ea37e1d3-a803-4f98-93d4-589ebbc59e42\" (UID: \"ea37e1d3-a803-4f98-93d4-589ebbc59e42\") "
Sep 15 19:28:25.456146 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.456048    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-metrics-certs\") pod \"15a6f1f4-2497-4960-9054-7275a473c292\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") "
Sep 15 19:28:25.456146 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.456077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a6f1f4-2497-4960-9054-7275a473c292-openshift-service-ca\") pod \"15a6f1f4-2497-4960-9054-7275a473c292\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") "
Sep 15 19:28:25.456146 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.456099    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhsqm\" (UniqueName: \"kubernetes.io/projected/15a6f1f4-2497-4960-9054-7275a473c292-kube-api-access-dhsqm\") pod \"15a6f1f4-2497-4960-9054-7275a473c292\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") "
Sep 15 19:28:25.456146 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.456141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-maas-api-tls\") pod \"15a6f1f4-2497-4960-9054-7275a473c292\" (UID: \"15a6f1f4-2497-4960-9054-7275a473c292\") "
Sep 15 19:28:25.456428 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.456393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea37e1d3-a803-4f98-93d4-589ebbc59e42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea37e1d3-a803-4f98-93d4-589ebbc59e42" (UID: "ea37e1d3-a803-4f98-93d4-589ebbc59e42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:25.456482 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.456441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a6f1f4-2497-4960-9054-7275a473c292-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15a6f1f4-2497-4960-9054-7275a473c292" (UID: "15a6f1f4-2497-4960-9054-7275a473c292"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:25.458365 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.458337    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea37e1d3-a803-4f98-93d4-589ebbc59e42-kube-api-access-9g8gh" (OuterVolumeSpecName: "kube-api-access-9g8gh") pod "ea37e1d3-a803-4f98-93d4-589ebbc59e42" (UID: "ea37e1d3-a803-4f98-93d4-589ebbc59e42"). InnerVolumeSpecName "kube-api-access-9g8gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:25.458492 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.458391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15a6f1f4-2497-4960-9054-7275a473c292" (UID: "15a6f1f4-2497-4960-9054-7275a473c292"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:28:25.458492 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.458394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a6f1f4-2497-4960-9054-7275a473c292-kube-api-access-dhsqm" (OuterVolumeSpecName: "kube-api-access-dhsqm") pod "15a6f1f4-2497-4960-9054-7275a473c292" (UID: "15a6f1f4-2497-4960-9054-7275a473c292"). InnerVolumeSpecName "kube-api-access-dhsqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:25.458492 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.458480    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "15a6f1f4-2497-4960-9054-7275a473c292" (UID: "15a6f1f4-2497-4960-9054-7275a473c292"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:28:25.556729 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.556703    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:25.556729 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.556725    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea37e1d3-a803-4f98-93d4-589ebbc59e42-plugins-config-volume\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:25.556729 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.556735    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g8gh\" (UniqueName: \"kubernetes.io/projected/ea37e1d3-a803-4f98-93d4-589ebbc59e42-kube-api-access-9g8gh\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:25.556938 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.556743    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a6f1f4-2497-4960-9054-7275a473c292-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:25.556938 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.556752    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a6f1f4-2497-4960-9054-7275a473c292-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:25.556938 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.556761    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhsqm\" (UniqueName: \"kubernetes.io/projected/15a6f1f4-2497-4960-9054-7275a473c292-kube-api-access-dhsqm\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:28:25.688355 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.688327    2578 generic.go:356] "Generic (PLEG): container finished" podID="ea37e1d3-a803-4f98-93d4-589ebbc59e42" containerID="a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd" exitCode=0
Sep 15 19:28:25.688504 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.688367    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"
Sep 15 19:28:25.688504 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.688410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh" event={"ID":"ea37e1d3-a803-4f98-93d4-589ebbc59e42","Type":"ContainerDied","Data":"a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd"}
Sep 15 19:28:25.688504 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.688457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh" event={"ID":"ea37e1d3-a803-4f98-93d4-589ebbc59e42","Type":"ContainerDied","Data":"16c87cea3256500f7240c655cb569aba283fbdb55bbad3347b4e61f3a0803326"}
Sep 15 19:28:25.688504 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.688491    2578 scope.go:117] "RemoveContainer" containerID="a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd"
Sep 15 19:28:25.689734 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.689708    2578 generic.go:356] "Generic (PLEG): container finished" podID="15a6f1f4-2497-4960-9054-7275a473c292" containerID="3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d" exitCode=0
Sep 15 19:28:25.689847 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.689747    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"
Sep 15 19:28:25.689847 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.689740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd" event={"ID":"15a6f1f4-2497-4960-9054-7275a473c292","Type":"ContainerDied","Data":"3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d"}
Sep 15 19:28:25.689847 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.689817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd" event={"ID":"15a6f1f4-2497-4960-9054-7275a473c292","Type":"ContainerDied","Data":"fddd2eaafc51ea621eb272d592d5edcd44d3b41d17b5e2f4c170912228f71746"}
Sep 15 19:28:25.698535 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.698517    2578 scope.go:117] "RemoveContainer" containerID="a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd"
Sep 15 19:28:25.698834 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:28:25.698792    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd\": container with ID starting with a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd not found: ID does not exist" containerID="a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd"
Sep 15 19:28:25.698913 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.698841    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd"} err="failed to get container status \"a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd\": rpc error: code = NotFound desc = could not find container \"a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd\": container with ID starting with a8d8325a38e795d18df2fa93f2e9e25c1757a4781ce32a65a5e4d20b7b674bdd not found: ID does not exist"
Sep 15 19:28:25.698913 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.698858    2578 scope.go:117] "RemoveContainer" containerID="3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d"
Sep 15 19:28:25.706638 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.706618    2578 scope.go:117] "RemoveContainer" containerID="3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d"
Sep 15 19:28:25.706928 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:28:25.706911    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d\": container with ID starting with 3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d not found: ID does not exist" containerID="3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d"
Sep 15 19:28:25.706986 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.706932    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d"} err="failed to get container status \"3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d\": rpc error: code = NotFound desc = could not find container \"3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d\": container with ID starting with 3ab2d2ae52ed4c4e8c4c66d86133ab8e48e2bd3a6b91476aa307cd45b813dc3d not found: ID does not exist"
Sep 15 19:28:25.736560 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.736533    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"]
Sep 15 19:28:25.758618 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.758599    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-415f87-866bb9b8d6-gfbvh"]
Sep 15 19:28:25.776639 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.776618    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"]
Sep 15 19:28:25.779856 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:25.779837    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-415f87-544fb59867-6wxwd"]
Sep 15 19:28:26.098398 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:26.098367    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a6f1f4-2497-4960-9054-7275a473c292" path="/var/lib/kubelet/pods/15a6f1f4-2497-4960-9054-7275a473c292/volumes"
Sep 15 19:28:26.098724 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:26.098711    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea37e1d3-a803-4f98-93d4-589ebbc59e42" path="/var/lib/kubelet/pods/ea37e1d3-a803-4f98-93d4-589ebbc59e42/volumes"
Sep 15 19:28:28.076332 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:28.076288    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:28:36.044455 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:28:36.044417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:29:56.074863 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:29:56.074776    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:30:00.129915 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.129878    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt"]
Sep 15 19:30:00.130366 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130280    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12e64d5-57a8-4045-9bd3-f840ef58a075" containerName="maas-api"
Sep 15 19:30:00.130366 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130299    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12e64d5-57a8-4045-9bd3-f840ef58a075" containerName="maas-api"
Sep 15 19:30:00.130366 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130314    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea37e1d3-a803-4f98-93d4-589ebbc59e42" containerName="payload-pre-processing"
Sep 15 19:30:00.130366 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130322    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea37e1d3-a803-4f98-93d4-589ebbc59e42" containerName="payload-pre-processing"
Sep 15 19:30:00.130366 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130333    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b8d06e-be59-4bbe-a6f1-ba443b3a4668" containerName="payload-processing"
Sep 15 19:30:00.130366 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130341    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b8d06e-be59-4bbe-a6f1-ba443b3a4668" containerName="payload-processing"
Sep 15 19:30:00.130366 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130353    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80222f3d-7154-4282-966f-f09402a4c66a" containerName="maas-api"
Sep 15 19:30:00.130366 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130360    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80222f3d-7154-4282-966f-f09402a4c66a" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130371    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81d4ecc-b9d7-403c-9af8-c396edd06ce8" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130379    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81d4ecc-b9d7-403c-9af8-c396edd06ce8" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130393    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a6f1f4-2497-4960-9054-7275a473c292" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130401    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a6f1f4-2497-4960-9054-7275a473c292" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130417    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa" containerName="payload-pre-processing"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130426    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa" containerName="payload-pre-processing"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130445    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c3ff10-076e-4c15-92cf-0843a7fcb89e" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130454    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c3ff10-076e-4c15-92cf-0843a7fcb89e" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130523    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea37e1d3-a803-4f98-93d4-589ebbc59e42" containerName="payload-pre-processing"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130535    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81d4ecc-b9d7-403c-9af8-c396edd06ce8" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130546    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12e64d5-57a8-4045-9bd3-f840ef58a075" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130555    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4b8d06e-be59-4bbe-a6f1-ba443b3a4668" containerName="payload-processing"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130566    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c3ff10-076e-4c15-92cf-0843a7fcb89e" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130574    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad802fa5-0cd3-4ad0-bfd3-49bdd88f2daa" containerName="payload-pre-processing"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130583    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a6f1f4-2497-4960-9054-7275a473c292" containerName="maas-api"
Sep 15 19:30:00.130783 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.130591    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="80222f3d-7154-4282-966f-f09402a4c66a" containerName="maas-api"
Sep 15 19:30:00.133594 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.133573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt"
Sep 15 19:30:00.135864 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.135841    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:30:00.139642 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.139620    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt"]
Sep 15 19:30:00.144759 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.144738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt"
Sep 15 19:30:00.411728 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:00.411699    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt"]
Sep 15 19:30:00.414194 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:30:00.414169    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8ee0c2_d6f4_4dcc_8d21_befc44181411.slice/crio-e49e209e5441d42e2bd6c097e4a3e5aa70bd4677333d41efa3e2a12d9dac0ab0 WatchSource:0}: Error finding container e49e209e5441d42e2bd6c097e4a3e5aa70bd4677333d41efa3e2a12d9dac0ab0: Status 404 returned error can't find the container with id e49e209e5441d42e2bd6c097e4a3e5aa70bd4677333d41efa3e2a12d9dac0ab0
Sep 15 19:30:01.100049 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:01.099960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt" event={"ID":"8f8ee0c2-d6f4-4dcc-8d21-befc44181411","Type":"ContainerStarted","Data":"65198b74043032905dfc6ed9e32694f3c54e1e283975ff6019825b243b03cf98"}
Sep 15 19:30:01.100049 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:01.100009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt" event={"ID":"8f8ee0c2-d6f4-4dcc-8d21-befc44181411","Type":"ContainerStarted","Data":"e49e209e5441d42e2bd6c097e4a3e5aa70bd4677333d41efa3e2a12d9dac0ab0"}
Sep 15 19:30:01.114485 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:01.114438    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt" podStartSLOduration=1.114423325 podStartE2EDuration="1.114423325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:30:01.112511835 +0000 UTC m=+2351.654607365" watchObservedRunningTime="2026-09-15 19:30:01.114423325 +0000 UTC m=+2351.656518872"
Sep 15 19:30:02.105917 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:02.105878    2578 generic.go:356] "Generic (PLEG): container finished" podID="8f8ee0c2-d6f4-4dcc-8d21-befc44181411" containerID="65198b74043032905dfc6ed9e32694f3c54e1e283975ff6019825b243b03cf98" exitCode=0
Sep 15 19:30:02.106397 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:02.105947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt" event={"ID":"8f8ee0c2-d6f4-4dcc-8d21-befc44181411","Type":"ContainerDied","Data":"65198b74043032905dfc6ed9e32694f3c54e1e283975ff6019825b243b03cf98"}
Sep 15 19:30:03.232640 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:03.232612    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt"
Sep 15 19:30:04.115117 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:04.115086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt" event={"ID":"8f8ee0c2-d6f4-4dcc-8d21-befc44181411","Type":"ContainerDied","Data":"e49e209e5441d42e2bd6c097e4a3e5aa70bd4677333d41efa3e2a12d9dac0ab0"}
Sep 15 19:30:04.115117 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:04.115106    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825010-v7nkt"
Sep 15 19:30:04.115298 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:04.115140    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e49e209e5441d42e2bd6c097e4a3e5aa70bd4677333d41efa3e2a12d9dac0ab0"
Sep 15 19:30:23.474716 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:23.474679    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:30:31.590025 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:31.589977    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:30:39.475664 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:39.475627    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:30:53.779747 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:30:53.779708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:31:04.386428 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:31:04.386390    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:32:31.980397 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:32:31.980362    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:32:39.078635 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:32:39.078595    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:32:48.574377 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:32:48.574295    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:32:56.781382 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:32:56.781344    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:33:11.006782 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.006748    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"]
Sep 15 19:33:11.007272 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.007082    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8ee0c2-d6f4-4dcc-8d21-befc44181411" containerName="cleanup"
Sep 15 19:33:11.007272 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.007102    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8ee0c2-d6f4-4dcc-8d21-befc44181411" containerName="cleanup"
Sep 15 19:33:11.007272 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.007206    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8ee0c2-d6f4-4dcc-8d21-befc44181411" containerName="cleanup"
Sep 15 19:33:11.010184 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.010160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.012944 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.012922    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b88ee092-kserve-self-signed-certs\""
Sep 15 19:33:11.020897 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.020869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"]
Sep 15 19:33:11.119432 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.119403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-dshm\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.119432 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.119432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tmp-dir\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.119625 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.119605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tls-certs\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.119665 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.119655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-home\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.119702 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.119679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9hcm\" (UniqueName: \"kubernetes.io/projected/e062fd0c-d1c2-4a6f-adf5-322968333c9e-kube-api-access-v9hcm\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.119848 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.119825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-model-cache\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.220782 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.220756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tls-certs\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.220936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.220814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-home\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.220936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.220832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9hcm\" (UniqueName: \"kubernetes.io/projected/e062fd0c-d1c2-4a6f-adf5-322968333c9e-kube-api-access-v9hcm\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.220936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.220850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-model-cache\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.220936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.220870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-dshm\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.220936 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.220885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tmp-dir\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.221208 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.221178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-home\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.221282 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.221258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-model-cache\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.221326 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.221289    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tmp-dir\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.223076 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.223056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-dshm\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.223335 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.223317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tls-certs\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.228771 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.228748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9hcm\" (UniqueName: \"kubernetes.io/projected/e062fd0c-d1c2-4a6f-adf5-322968333c9e-kube-api-access-v9hcm\") pod \"e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") " pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.319227 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.319169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:11.594368 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.594326    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"]
Sep 15 19:33:11.596751 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:33:11.596724    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode062fd0c_d1c2_4a6f_adf5_322968333c9e.slice/crio-86ffad58674786a71581ee70eb6d3c45408d0b7d7bbf2505120ebfd98551475a WatchSource:0}: Error finding container 86ffad58674786a71581ee70eb6d3c45408d0b7d7bbf2505120ebfd98551475a: Status 404 returned error can't find the container with id 86ffad58674786a71581ee70eb6d3c45408d0b7d7bbf2505120ebfd98551475a
Sep 15 19:33:11.887333 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.887299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4" event={"ID":"e062fd0c-d1c2-4a6f-adf5-322968333c9e","Type":"ContainerStarted","Data":"e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762"}
Sep 15 19:33:11.887507 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.887343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4" event={"ID":"e062fd0c-d1c2-4a6f-adf5-322968333c9e","Type":"ContainerStarted","Data":"86ffad58674786a71581ee70eb6d3c45408d0b7d7bbf2505120ebfd98551475a"}
Sep 15 19:33:11.905814 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:11.905752    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4" podStartSLOduration=1.9057357719999999 podStartE2EDuration="1.905735772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:33:11.904449589 +0000 UTC m=+2542.446545117" watchObservedRunningTime="2026-09-15 19:33:11.905735772 +0000 UTC m=+2542.447831295"
Sep 15 19:33:14.078424 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:14.078387    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:33:21.320195 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:21.320160    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:21.320658 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:21.320210    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:21.332714 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:21.332685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:21.938888 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:21.938860    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:43.211661 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:43.211626    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"]
Sep 15 19:33:45.386589 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:45.386546    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:33:53.178622 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:53.178579    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:33:58.231943 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.231885    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4" podUID="e062fd0c-d1c2-4a6f-adf5-322968333c9e" containerName="main" containerID="cri-o://e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762" gracePeriod=45
Sep 15 19:33:58.477152 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.477130    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:58.509620 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.509537    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tls-certs\") pod \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") "
Sep 15 19:33:58.509620 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.509580    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-model-cache\") pod \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") "
Sep 15 19:33:58.509857 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.509620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-home\") pod \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") "
Sep 15 19:33:58.509857 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.509652    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9hcm\" (UniqueName: \"kubernetes.io/projected/e062fd0c-d1c2-4a6f-adf5-322968333c9e-kube-api-access-v9hcm\") pod \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") "
Sep 15 19:33:58.509857 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.509680    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tmp-dir\") pod \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") "
Sep 15 19:33:58.509857 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.509703    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-dshm\") pod \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\" (UID: \"e062fd0c-d1c2-4a6f-adf5-322968333c9e\") "
Sep 15 19:33:58.510055 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.509906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e062fd0c-d1c2-4a6f-adf5-322968333c9e" (UID: "e062fd0c-d1c2-4a6f-adf5-322968333c9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:33:58.510055 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.509936    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-home" (OuterVolumeSpecName: "home") pod "e062fd0c-d1c2-4a6f-adf5-322968333c9e" (UID: "e062fd0c-d1c2-4a6f-adf5-322968333c9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:33:58.510055 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.509965    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e062fd0c-d1c2-4a6f-adf5-322968333c9e" (UID: "e062fd0c-d1c2-4a6f-adf5-322968333c9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:33:58.511863 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.511834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e062fd0c-d1c2-4a6f-adf5-322968333c9e" (UID: "e062fd0c-d1c2-4a6f-adf5-322968333c9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:33:58.511981 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.511865    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-dshm" (OuterVolumeSpecName: "dshm") pod "e062fd0c-d1c2-4a6f-adf5-322968333c9e" (UID: "e062fd0c-d1c2-4a6f-adf5-322968333c9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:33:58.511981 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.511875    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e062fd0c-d1c2-4a6f-adf5-322968333c9e-kube-api-access-v9hcm" (OuterVolumeSpecName: "kube-api-access-v9hcm") pod "e062fd0c-d1c2-4a6f-adf5-322968333c9e" (UID: "e062fd0c-d1c2-4a6f-adf5-322968333c9e"). InnerVolumeSpecName "kube-api-access-v9hcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:33:58.610911 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.610880    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-home\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:33:58.610911 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.610905    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9hcm\" (UniqueName: \"kubernetes.io/projected/e062fd0c-d1c2-4a6f-adf5-322968333c9e-kube-api-access-v9hcm\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:33:58.610911 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.610915    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tmp-dir\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:33:58.611136 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.610926    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-dshm\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:33:58.611136 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.610934    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e062fd0c-d1c2-4a6f-adf5-322968333c9e-tls-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:33:58.611136 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:58.610941    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e062fd0c-d1c2-4a6f-adf5-322968333c9e-model-cache\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:33:59.084786 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:59.084750    2578 generic.go:356] "Generic (PLEG): container finished" podID="e062fd0c-d1c2-4a6f-adf5-322968333c9e" containerID="e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762" exitCode=0
Sep 15 19:33:59.084786 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:59.084786    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"
Sep 15 19:33:59.085007 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:59.084823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4" event={"ID":"e062fd0c-d1c2-4a6f-adf5-322968333c9e","Type":"ContainerDied","Data":"e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762"}
Sep 15 19:33:59.085007 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:59.084857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4" event={"ID":"e062fd0c-d1c2-4a6f-adf5-322968333c9e","Type":"ContainerDied","Data":"86ffad58674786a71581ee70eb6d3c45408d0b7d7bbf2505120ebfd98551475a"}
Sep 15 19:33:59.085007 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:59.084878    2578 scope.go:117] "RemoveContainer" containerID="e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762"
Sep 15 19:33:59.093649 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:59.093628    2578 scope.go:117] "RemoveContainer" containerID="e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762"
Sep 15 19:33:59.093941 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:33:59.093925    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762\": container with ID starting with e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762 not found: ID does not exist" containerID="e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762"
Sep 15 19:33:59.094000 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:59.093949    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762"} err="failed to get container status \"e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762\": rpc error: code = NotFound desc = could not find container \"e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762\": container with ID starting with e83e03118d0d2eb22ef2f244f471b6ec453bf3abb9473046b27b7ef421e6d762 not found: ID does not exist"
Sep 15 19:33:59.104033 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:59.104004    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"]
Sep 15 19:33:59.108219 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:33:59.108197    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b88ee092-kserve-6b69dd9979-vsdf4"]
Sep 15 19:34:00.099203 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.099167    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e062fd0c-d1c2-4a6f-adf5-322968333c9e" path="/var/lib/kubelet/pods/e062fd0c-d1c2-4a6f-adf5-322968333c9e/volumes"
Sep 15 19:34:00.385066 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.385032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:34:00.515623 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.515585    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"]
Sep 15 19:34:00.515899 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.515885    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e062fd0c-d1c2-4a6f-adf5-322968333c9e" containerName="main"
Sep 15 19:34:00.515899 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.515899    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e062fd0c-d1c2-4a6f-adf5-322968333c9e" containerName="main"
Sep 15 19:34:00.515986 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.515962    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e062fd0c-d1c2-4a6f-adf5-322968333c9e" containerName="main"
Sep 15 19:34:00.520142 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.520125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.523638 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.523605    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:34:00.523638 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.523616    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 19:34:00.523638 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.523606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 19:34:00.523916 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.523659    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v55qf\""
Sep 15 19:34:00.523916 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.523643    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 19:34:00.530253 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.530228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"]
Sep 15 19:34:00.531924 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.531903    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"]
Sep 15 19:34:00.535095 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.535074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.545604 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.545567    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"]
Sep 15 19:34:00.624890 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.624843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-openshift-service-ca\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.624890 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.624888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-maas-api-tls\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.625174 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.624932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-maas-api-tls\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.625174 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.624962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-metrics-certs\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.625174 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.624983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe68388b-9cf8-4b87-8105-ef529cfb8152-openshift-service-ca\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.625174 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.625001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4zrc\" (UniqueName: \"kubernetes.io/projected/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-kube-api-access-g4zrc\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.625174 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.625023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-metrics-certs\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.625174 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.625051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6q9\" (UniqueName: \"kubernetes.io/projected/fe68388b-9cf8-4b87-8105-ef529cfb8152-kube-api-access-sh6q9\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.726277 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.726172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-openshift-service-ca\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.726277 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.726223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-maas-api-tls\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.726277 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.726249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-maas-api-tls\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.726277 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.726279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-metrics-certs\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.726554 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.726450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe68388b-9cf8-4b87-8105-ef529cfb8152-openshift-service-ca\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.726554 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.726500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zrc\" (UniqueName: \"kubernetes.io/projected/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-kube-api-access-g4zrc\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.726554 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.726543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-metrics-certs\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.726697 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.726566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6q9\" (UniqueName: \"kubernetes.io/projected/fe68388b-9cf8-4b87-8105-ef529cfb8152-kube-api-access-sh6q9\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.727122 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.727093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-openshift-service-ca\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.727280 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.727256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe68388b-9cf8-4b87-8105-ef529cfb8152-openshift-service-ca\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.728925 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.728902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-metrics-certs\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.729048 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.728962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-maas-api-tls\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.729218 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.729198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-metrics-certs\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.729365 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.729346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-maas-api-tls\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.734003 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.733974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6q9\" (UniqueName: \"kubernetes.io/projected/fe68388b-9cf8-4b87-8105-ef529cfb8152-kube-api-access-sh6q9\") pod \"maas-api-59f6566bbc-sxfmz\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.734003 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.733987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zrc\" (UniqueName: \"kubernetes.io/projected/92ab43f6-0b25-4c8b-a3af-4ac55b176f1b-kube-api-access-g4zrc\") pod \"maas-api-86d9ddc875-4gg2z\" (UID: \"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:00.832110 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.832072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:00.846288 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:00.846258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:01.131241 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:01.131214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"]
Sep 15 19:34:01.133628 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:34:01.133604    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe68388b_9cf8_4b87_8105_ef529cfb8152.slice/crio-df885af95af0a6c252ac2ea6a357eef40340ac56b7c825eda5659e2f4cc1f02f WatchSource:0}: Error finding container df885af95af0a6c252ac2ea6a357eef40340ac56b7c825eda5659e2f4cc1f02f: Status 404 returned error can't find the container with id df885af95af0a6c252ac2ea6a357eef40340ac56b7c825eda5659e2f4cc1f02f
Sep 15 19:34:01.134879 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:01.134861    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:34:01.159916 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:01.159838    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"]
Sep 15 19:34:01.161779 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:34:01.161752    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ab43f6_0b25_4c8b_a3af_4ac55b176f1b.slice/crio-4fbf65eb301ead9f27130e5a4798fd918c3ce78c17b7262b6c92af80fa1a05bf WatchSource:0}: Error finding container 4fbf65eb301ead9f27130e5a4798fd918c3ce78c17b7262b6c92af80fa1a05bf: Status 404 returned error can't find the container with id 4fbf65eb301ead9f27130e5a4798fd918c3ce78c17b7262b6c92af80fa1a05bf
Sep 15 19:34:02.099714 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:02.099679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z" event={"ID":"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b","Type":"ContainerStarted","Data":"5bb660fe562f1092c63855e280a95da4715537dc46875056bbb4e400341ab5a3"}
Sep 15 19:34:02.099917 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:02.099722    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z" event={"ID":"92ab43f6-0b25-4c8b-a3af-4ac55b176f1b","Type":"ContainerStarted","Data":"4fbf65eb301ead9f27130e5a4798fd918c3ce78c17b7262b6c92af80fa1a05bf"}
Sep 15 19:34:02.099917 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:02.099764    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:02.101166 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:02.101140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz" event={"ID":"fe68388b-9cf8-4b87-8105-ef529cfb8152","Type":"ContainerStarted","Data":"028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79"}
Sep 15 19:34:02.101253 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:02.101187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz" event={"ID":"fe68388b-9cf8-4b87-8105-ef529cfb8152","Type":"ContainerStarted","Data":"df885af95af0a6c252ac2ea6a357eef40340ac56b7c825eda5659e2f4cc1f02f"}
Sep 15 19:34:02.101253 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:02.101216    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:02.119351 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:02.117014    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z" podStartSLOduration=1.726661046 podStartE2EDuration="2.116994977s" totalImagesPullingTime="390.333931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:34:02.116332924 +0000 UTC m=+2592.658428455" watchObservedRunningTime="2026-09-15 19:34:02.116994977 +0000 UTC m=+2592.659090506"
Sep 15 19:34:02.139064 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:02.138992    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz" podStartSLOduration=1.6390020939999999 podStartE2EDuration="2.138973206s" totalImagesPullingTime="499.971112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:34:02.137449635 +0000 UTC m=+2592.679545166" watchObservedRunningTime="2026-09-15 19:34:02.138973206 +0000 UTC m=+2592.681068735"
Sep 15 19:34:07.977024 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:07.976985    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:34:08.112185 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.112154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-4gg2z"
Sep 15 19:34:08.112361 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.112323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:08.174378 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.174342    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"]
Sep 15 19:34:08.174578 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.174557    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz" podUID="fe68388b-9cf8-4b87-8105-ef529cfb8152" containerName="maas-api" containerID="cri-o://028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79" gracePeriod=30
Sep 15 19:34:08.414137 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.414108    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:08.497966 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.497927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-metrics-certs\") pod \"fe68388b-9cf8-4b87-8105-ef529cfb8152\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") "
Sep 15 19:34:08.497966 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.497968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe68388b-9cf8-4b87-8105-ef529cfb8152-openshift-service-ca\") pod \"fe68388b-9cf8-4b87-8105-ef529cfb8152\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") "
Sep 15 19:34:08.498199 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.498040    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh6q9\" (UniqueName: \"kubernetes.io/projected/fe68388b-9cf8-4b87-8105-ef529cfb8152-kube-api-access-sh6q9\") pod \"fe68388b-9cf8-4b87-8105-ef529cfb8152\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") "
Sep 15 19:34:08.498199 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.498060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-maas-api-tls\") pod \"fe68388b-9cf8-4b87-8105-ef529cfb8152\" (UID: \"fe68388b-9cf8-4b87-8105-ef529cfb8152\") "
Sep 15 19:34:08.498381 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.498357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe68388b-9cf8-4b87-8105-ef529cfb8152-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe68388b-9cf8-4b87-8105-ef529cfb8152" (UID: "fe68388b-9cf8-4b87-8105-ef529cfb8152"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:34:08.500163 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.500138    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe68388b-9cf8-4b87-8105-ef529cfb8152" (UID: "fe68388b-9cf8-4b87-8105-ef529cfb8152"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:34:08.500247 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.500154    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fe68388b-9cf8-4b87-8105-ef529cfb8152" (UID: "fe68388b-9cf8-4b87-8105-ef529cfb8152"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:34:08.500286 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.500250    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe68388b-9cf8-4b87-8105-ef529cfb8152-kube-api-access-sh6q9" (OuterVolumeSpecName: "kube-api-access-sh6q9") pod "fe68388b-9cf8-4b87-8105-ef529cfb8152" (UID: "fe68388b-9cf8-4b87-8105-ef529cfb8152"). InnerVolumeSpecName "kube-api-access-sh6q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:34:08.599486 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.599385    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6q9\" (UniqueName: \"kubernetes.io/projected/fe68388b-9cf8-4b87-8105-ef529cfb8152-kube-api-access-sh6q9\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:34:08.599486 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.599426    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-maas-api-tls\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:34:08.599486 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.599441    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe68388b-9cf8-4b87-8105-ef529cfb8152-metrics-certs\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:34:08.599486 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:08.599451    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe68388b-9cf8-4b87-8105-ef529cfb8152-openshift-service-ca\") on node \"ip-10-0-135-221.ec2.internal\" DevicePath \"\""
Sep 15 19:34:09.132657 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:09.132615    2578 generic.go:356] "Generic (PLEG): container finished" podID="fe68388b-9cf8-4b87-8105-ef529cfb8152" containerID="028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79" exitCode=0
Sep 15 19:34:09.133076 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:09.132663    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"
Sep 15 19:34:09.133076 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:09.132700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz" event={"ID":"fe68388b-9cf8-4b87-8105-ef529cfb8152","Type":"ContainerDied","Data":"028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79"}
Sep 15 19:34:09.133076 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:09.132758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz" event={"ID":"fe68388b-9cf8-4b87-8105-ef529cfb8152","Type":"ContainerDied","Data":"df885af95af0a6c252ac2ea6a357eef40340ac56b7c825eda5659e2f4cc1f02f"}
Sep 15 19:34:09.133076 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:09.132782    2578 scope.go:117] "RemoveContainer" containerID="028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79"
Sep 15 19:34:09.141526 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:09.141508    2578 scope.go:117] "RemoveContainer" containerID="028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79"
Sep 15 19:34:09.141778 ip-10-0-135-221 kubenswrapper[2578]: E0915 19:34:09.141762    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79\": container with ID starting with 028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79 not found: ID does not exist" containerID="028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79"
Sep 15 19:34:09.141849 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:09.141785    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79"} err="failed to get container status \"028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79\": rpc error: code = NotFound desc = could not find container \"028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79\": container with ID starting with 028930d882df696141081fce305809c6a145ef818c57133aad4db10f349a1e79 not found: ID does not exist"
Sep 15 19:34:09.152344 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:09.152303    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"]
Sep 15 19:34:09.155626 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:09.155603    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-sxfmz"]
Sep 15 19:34:10.097888 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:10.097852    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe68388b-9cf8-4b87-8105-ef529cfb8152" path="/var/lib/kubelet/pods/fe68388b-9cf8-4b87-8105-ef529cfb8152/volumes"
Sep 15 19:34:14.478440 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:14.478219    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:34:19.675492 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:19.675453    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:34:25.980702 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:25.980663    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfp68"]
Sep 15 19:34:32.016133 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:32.016100    2578 ???:1] "http: TLS handshake error from 10.0.140.183:60128: EOF"
Sep 15 19:34:32.023575 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:32.023537    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9qhdm_e3ef2522-f438-42f8-a961-b5b62e5c4836/global-pull-secret-syncer/0.log"
Sep 15 19:34:32.144620 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:32.144584    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-srpw5_b251128d-040c-49f9-8396-36a2db07a539/konnectivity-agent/0.log"
Sep 15 19:34:32.202157 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:32.202125    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-221.ec2.internal_1f58418f2eb3a96678dd7e1f043ca38f/haproxy/0.log"
Sep 15 19:34:36.718050 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:36.718019    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dfp68_f96f6a13-88c8-4a17-a26f-b665fc04e950/limitador/0.log"
Sep 15 19:34:38.604453 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:38.604423    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7r5v6_ef6be42b-32b1-4166-bae4-ea69002cb587/node-exporter/0.log"
Sep 15 19:34:38.630270 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:38.630244    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7r5v6_ef6be42b-32b1-4166-bae4-ea69002cb587/kube-rbac-proxy/0.log"
Sep 15 19:34:38.651570 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:38.651549    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7r5v6_ef6be42b-32b1-4166-bae4-ea69002cb587/init-textfile/0.log"
Sep 15 19:34:39.196293 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:39.196261    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-2wcdq_1cec380a-fe2b-4919-b539-de206b422c84/prometheus-operator-admission-webhook/0.log"
Sep 15 19:34:39.328144 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:39.328099    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57875dc5d5-s4gxw_b1c57538-53ad-4a20-aaa9-21ce2a5e70c4/thanos-query/0.log"
Sep 15 19:34:39.351235 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:39.351162    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57875dc5d5-s4gxw_b1c57538-53ad-4a20-aaa9-21ce2a5e70c4/kube-rbac-proxy-web/0.log"
Sep 15 19:34:39.372289 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:39.372266    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57875dc5d5-s4gxw_b1c57538-53ad-4a20-aaa9-21ce2a5e70c4/kube-rbac-proxy/0.log"
Sep 15 19:34:39.396309 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:39.396286    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57875dc5d5-s4gxw_b1c57538-53ad-4a20-aaa9-21ce2a5e70c4/prom-label-proxy/0.log"
Sep 15 19:34:39.417249 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:39.417224    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57875dc5d5-s4gxw_b1c57538-53ad-4a20-aaa9-21ce2a5e70c4/kube-rbac-proxy-rules/0.log"
Sep 15 19:34:39.441198 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:39.441172    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57875dc5d5-s4gxw_b1c57538-53ad-4a20-aaa9-21ce2a5e70c4/kube-rbac-proxy-metrics/0.log"
Sep 15 19:34:42.277432 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.277391    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"]
Sep 15 19:34:42.277855 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.277688    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe68388b-9cf8-4b87-8105-ef529cfb8152" containerName="maas-api"
Sep 15 19:34:42.277855 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.277700    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe68388b-9cf8-4b87-8105-ef529cfb8152" containerName="maas-api"
Sep 15 19:34:42.277855 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.277761    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe68388b-9cf8-4b87-8105-ef529cfb8152" containerName="maas-api"
Sep 15 19:34:42.280594 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.280574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.282989 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.282962    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xxzwt\"/\"openshift-service-ca.crt\""
Sep 15 19:34:42.283111 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.282985    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xxzwt\"/\"kube-root-ca.crt\""
Sep 15 19:34:42.283760 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.283744    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xxzwt\"/\"default-dockercfg-7hxc4\""
Sep 15 19:34:42.291338 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.291315    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"]
Sep 15 19:34:42.383367 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.383330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-proc\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.383367 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.383364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-sys\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.383581 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.383460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-lib-modules\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.383581 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.383496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-podres\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.383581 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.383516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrfj\" (UniqueName: \"kubernetes.io/projected/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-kube-api-access-fdrfj\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.484898 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.484862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-lib-modules\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.484898 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.484900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-podres\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.485092 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.484920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrfj\" (UniqueName: \"kubernetes.io/projected/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-kube-api-access-fdrfj\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.485092 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.484946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-proc\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.485092 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.484964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-sys\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.485092 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.485034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-sys\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.485092 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.485042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-podres\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.485092 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.485042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-lib-modules\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.485092 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.485073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-proc\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.494096 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.494076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrfj\" (UniqueName: \"kubernetes.io/projected/5abff8d6-d56d-42fb-a94c-d3c1462ffb4b-kube-api-access-fdrfj\") pod \"perf-node-gather-daemonset-6mlrp\" (UID: \"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.590698 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.590616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:42.865767 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:42.862854    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"]
Sep 15 19:34:42.869167 ip-10-0-135-221 kubenswrapper[2578]: W0915 19:34:42.869138    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5abff8d6_d56d_42fb_a94c_d3c1462ffb4b.slice/crio-ed7adfbc8f8e2da3f4cc14a93a8bcebc9de660aed7d537bc9bfe3df085f57e70 WatchSource:0}: Error finding container ed7adfbc8f8e2da3f4cc14a93a8bcebc9de660aed7d537bc9bfe3df085f57e70: Status 404 returned error can't find the container with id ed7adfbc8f8e2da3f4cc14a93a8bcebc9de660aed7d537bc9bfe3df085f57e70
Sep 15 19:34:43.279255 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:43.279218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp" event={"ID":"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b","Type":"ContainerStarted","Data":"bf65e873da6ff5c583cce25f32ed0d35b95ceb3cf1f2ca4bea2cb4af94e8a04f"}
Sep 15 19:34:43.279640 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:43.279282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp" event={"ID":"5abff8d6-d56d-42fb-a94c-d3c1462ffb4b","Type":"ContainerStarted","Data":"ed7adfbc8f8e2da3f4cc14a93a8bcebc9de660aed7d537bc9bfe3df085f57e70"}
Sep 15 19:34:43.279640 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:43.279310    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:43.300088 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:43.300042    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp" podStartSLOduration=1.300028411 podStartE2EDuration="1.300028411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 19:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:34:43.298929829 +0000 UTC m=+2633.841025358" watchObservedRunningTime="2026-09-15 19:34:43.300028411 +0000 UTC m=+2633.842123940"
Sep 15 19:34:44.379126 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:44.379099    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bngct_05d53002-a924-4ea1-a656-da4e6ba1e6db/dns/0.log"
Sep 15 19:34:44.409460 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:44.409429    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bngct_05d53002-a924-4ea1-a656-da4e6ba1e6db/kube-rbac-proxy/0.log"
Sep 15 19:34:44.489601 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:44.489569    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4gljt_06644277-8f48-4af9-930e-c6493a57b420/dns-node-resolver/0.log"
Sep 15 19:34:45.113838 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:45.113792    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4glsv_ac29fcda-9928-4ad9-9cee-6456cabf59a0/node-ca/0.log"
Sep 15 19:34:46.462456 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:46.462418    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8hhr7_f90aab5f-4144-413c-a9b2-0ce2c73f41da/discovery/0.log"
Sep 15 19:34:47.550067 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:47.550038    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bhb6b_9cd544b8-ee8e-4117-9103-d67234994f82/serve-healthcheck-canary/0.log"
Sep 15 19:34:48.143833 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:48.143792    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hn444_3ac0fda6-2405-4bcf-a74b-612af3c2268c/kube-rbac-proxy/0.log"
Sep 15 19:34:48.164582 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:48.164552    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hn444_3ac0fda6-2405-4bcf-a74b-612af3c2268c/exporter/0.log"
Sep 15 19:34:48.188767 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:48.188739    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hn444_3ac0fda6-2405-4bcf-a74b-612af3c2268c/extractor/0.log"
Sep 15 19:34:49.293086 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:49.293055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-6mlrp"
Sep 15 19:34:50.707463 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:50.707434    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-27p29_2baaef58-5aa5-465e-8261-e711db65b18b/server/0.log"
Sep 15 19:34:50.817383 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:50.817356    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d9df7495-cl6n5_9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4/manager/0.log"
Sep 15 19:34:50.838901 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:50.838879    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d9df7495-cl6n5_9ff8d4ea-79aa-48c2-aaf0-c7cc346bfff4/copy-manifests/0.log"
Sep 15 19:34:52.541584 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:52.541557    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5bcb5dcf99-fjjtm_1dbf67ad-ec58-489c-a0f4-d6cdfee5cc69/manager/0.log"
Sep 15 19:34:53.047706 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:53.047673    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8hhr7_f90aab5f-4144-413c-a9b2-0ce2c73f41da/discovery/0.log"
Sep 15 19:34:54.945013 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:54.944981    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8hhr7_f90aab5f-4144-413c-a9b2-0ce2c73f41da/discovery/0.log"
Sep 15 19:34:59.014224 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:59.014190    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9nr27_2845410e-10e0-4248-862b-1d7bed1fd00d/kube-multus/0.log"
Sep 15 19:34:59.038505 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:59.038482    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b75vj_30cfe649-c99b-45ad-9325-c187e24073c3/kube-multus-additional-cni-plugins/0.log"
Sep 15 19:34:59.059381 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:59.059355    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b75vj_30cfe649-c99b-45ad-9325-c187e24073c3/egress-router-binary-copy/0.log"
Sep 15 19:34:59.080009 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:59.079995    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b75vj_30cfe649-c99b-45ad-9325-c187e24073c3/cni-plugins/0.log"
Sep 15 19:34:59.099454 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:59.099436    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b75vj_30cfe649-c99b-45ad-9325-c187e24073c3/bond-cni-plugin/0.log"
Sep 15 19:34:59.123022 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:59.123007    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b75vj_30cfe649-c99b-45ad-9325-c187e24073c3/routeoverride-cni/0.log"
Sep 15 19:34:59.144055 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:59.144025    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b75vj_30cfe649-c99b-45ad-9325-c187e24073c3/whereabouts-cni-bincopy/0.log"
Sep 15 19:34:59.165921 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:59.165901    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b75vj_30cfe649-c99b-45ad-9325-c187e24073c3/whereabouts-cni/0.log"
Sep 15 19:34:59.638578 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:59.638555    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hktjq_eb8d5cee-7842-4f44-9079-94434a03ccff/network-metrics-daemon/0.log"
Sep 15 19:34:59.660218 ip-10-0-135-221 kubenswrapper[2578]: I0915 19:34:59.660167    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hktjq_eb8d5cee-7842-4f44-9079-94434a03ccff/kube-rbac-proxy/0.log"